Indexer Failed: Mapper Parsing Expection -> failed to parse Channel "Security" as number

[theSpawn (Michael kahr)]

Hi

we use NXLOG Enterprise with NXLog Manager since a few weeks and all worked fine, until yesterday.

we rebooted the machine and face since that “mapper parsing exceptions” in graylog.

All events from our domain controllers create the following events:

{“type”:“mapper_parsing_exception”,“reason”:“failed to parse [Channel]”,“caused_by”:{“type”:“number_format_exception”,“reason”:“For input string: “Security””}}

{"type":"mapper_parsing_exception","reason":"failed to parse [Channel]","caused_by":{"type":"number_format_exception","reason":"For input string: \"Microsoft-Windows-WMI-Activity/Operational\""}}

Except DNS. DNS Events are still parsable by Graylog/ElasticSearch.

I’ve already contacted NXLOG and they told me, that the problem itself is graylog, as it misinterprets the sent data?!?

I’ve checked the forum already and found multiple entries about similar problems with different solutions.

one solution is to create special elastic-search configs, but thats one thing, which I can’t follow up, as the parser worked already… , the other Solution was to install ElasticSearch 5, but I can’t find any documentation, that that would work.

we are currently running on:

elasticsearch/unbekannt,now 2.4.6 all
graylog-server/stable,stable,now 2.4.6-1 all / graylog-2.4-repository/stable,stable,now 1-6 all

Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

Linux 4.4.0-135-generic #161-Ubuntu SMP Mon Aug 27 10:45:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Input:

GELF TCP / 12201 without SSL / No Extractors or simliar

We have only the default GROK-Patterns in use, which were shipped, nothing custom at all.

do you have any hints, how we can get the parser working again?

regards michael

[jan (Jan Doberstein)]

the problem is, that elasticsearch tries to guess what data is ingested by the first data that is used to create the field.

After the index rotation, this process starts new for each field - as long as you do not have a custom index mapping that forces Elasticsearch to use a specific datatype for one field.

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

You could try with a forced index rotate first to see if the error is fixed, but I guess that the Data in the DNS events is something different than from all other sources.

[theSpawn (Michael kahr)]

Hi Jan

thanks for the reply. will try with the rotate on all my indexes and give feedback.

I still don’t understand, why it worked with the parser and after a reboot, it won’t work again…

michael

[theSpawn (Michael kahr)]

Hi

the rotate of the index is now completly done and the are re-occuring again.

in the “Stream-Overview”, I can see that in our Stream for the Windows Events are “3877 messages/second” shown, but once I open the stream, there are only events from Windows DNS visible…

I have now created a new index file and stream for DNS Events and route them via “SourceModuleName = DNSLOG” into the DNS-Stream.

I will now retrigger the rotate again and check, if the problem is fixed afterwards.

due investigating some log-files, I found the main issue.

in Windows-DNS-Events is Channel a number (17 for example), but for all other Events it’s an alphabetics like Security or similar

regards michael

[jan (Jan Doberstein)]

he @theSpawn

you could go and create an index mapping that forces the field to be a string - to prevent that failure in the future or you create one index dedicated to DNS …

[theSpawn (Michael kahr)]

Hi

if I want to force it to a string, would that be a custom elastic search rule as mentioned before or something in graylog itself?

regards michael

[jan (Jan Doberstein)]

To force this field to be a string, you would create a custom elasticsearch mapping like explained in the documentation:

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

[theSpawn (Michael kahr)]

Hi

I fixed the issue!

Root-Problem: my colleagues configured NXLOG Manager to send Events from “ms_vistalog” and from “im_etw” in one “path”.

ms_vistalog is default windows event log and the im_etw is the analytical dns.

the format of the events are slightly different:

ms_vistalog has as Channel the Log-Part fE " Security" or “Windows PowerShell”.
im_etw has as Channel “16” or “17”, depending on the Event-Type

as this was sent in one “path”, elasticsearch got the parser-issues.

solution:

split the Config in two Paths and send the DNS_Events (im_etw) via a separate Port to separate Streams / Indices.

NXLOG:
Stream-Rule: Domain-Controllers:

• Path1: Windows Event Logs -> TCP OUT GELF 12201
• Path 2: Windows DNS Logs -> TCP OUT GELF 12202

Graylog:

Part 1:

• Input: Windows Event Log (GELF TCP Port 12201)
• Stream: Windows Event Log
• Stream-Rule: gl2_input_source -> to the Windows Event Log GELF Input
• Indices: Win_AD_Events

Part 2:

• Input Windows DNS Events (GELF TCP Port 12202)
• Stream: Windows DNS Log
• Stream-Rule: gl2_input_source -> to the Windows DNS GELF Input
• Indices: Win_DNS_Events

then we forced a manually rotation of the indices.

once we did and enforced the rotation of the incides, the errors have disappeared and the logs are parsed perfectly.

I will forward that to NX_Log Support, as in my oppinion, that’s their part of support.

thanks for your support and fast reactions!

Regards Michael