Winlogbeat templates not included?

[Totally_Not_A_Robot (Tess)]

Hey guys,

I’m mostly rubber-duckying here, apologies if the thread is mostly me answering my own questions…

Situation
Last week our Graylog cluster was puttering along just fine, everything fine and dandy. This morning, I come in to find no less than 38k indexer failures on the System > Overview.

Troubleshooting
The indexer failures details tell me that all of the errors revolve around one issue:

type: mapper_parsing_exception, reason: failed to parse [winlogbeat_event_data_param2], cause by: {type:illegal_argument_exception, reason: invalid format}

Looking at the Graylog statistics dashboard I made for my team, I see that Graylog was receiving ~48k messages per hour until Sunday 00:00. After that it exploded into ~110k an hour.

To me this suggests that my indexes rolled over on Sunday morning, resetting the field types in the index after which ElasticSearch would start dynamic mapping anew. @jochen’s reply in the thread linked below have been very helpful in coming to that conclusion.

Graylog Indexer Failure On Exchange Logs

I have recently started getting the following Graylog Indexer Failures. I know I need to make a custom mapping but I can’t find the custom mapping file. Where is it located. I would also like to get help writing my mapping before applying it. exchange_index_1 {“type”:“mapper_parsing_exception”,“reason”:“failed to parse [message_info]”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“Invalid format: “04I:” is malformed at “I:””}} I can’t quite figure out what my mapping.json file s…

Hypothesis
Elasticsearch does not appear to have the Winlogbeat templates pre-loaded. This idea is further solidified by checking the following, which only shows the default Graylog templates:

curl -u $GRAYLOGUSER --cacert $CACERT https://elastichost:9200/_template/?pretty

I don’t know why the Winlogbeat template isn’t in there because I had assume that Graylog would load that in because Graylog comes with Winlogbeat receivers by default. I had assumed that setting up Graylog would also make sure that Elastic knew what Graylog’s default supported output means.

But you know what they say about assumptions.

I’ll be back later… First I’ll try to figure out where to get a recent Winlogbeat template and how to fudge it into Elastic.

[jan (Jan Doberstein)]

your conclusion is correct, and the solution you are searching is here: http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

In addition I would like to name cerebro ( https://github.com/lmenezes/cerebro ) as one of the best elasticsearch frontends I have seen.

[Totally_Not_A_Robot (Tess)]

@jan

your conclusion is correct, and the solution you are searching is here: http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

After some digging I have found “winlogbeat.template.json” which apparently is part of the Winlogbeat distribution from the Elastic people.

https://raw.githubusercontent.com/elastic/beats/v5.5.2/winlogbeat/winlogbeat.template.json

I’m currently poking through @jochen’s post to figure out how to load it using curl. I hadn’t counted on having to manage Elastic beyond the basic clustering, so this is new ground It’s not unenjoyable, but a bit unexpected.

EDIT
Right, loading the template was as easy as following the one-liner in the Graylog doc you linked.

curl -X PUT -d @'winlogbeat.template.json' -u $GRAYLOGUSER --cacert $CACERT https://elastichost:9200/_template/winlogbeat/?pretty

Now I just have to figure out how to trigger another index roll-over

EDIT
Found that as well Manually rotating the write index.

[Totally_Not_A_Robot (Tess)]

Right… I’ve loaded the template on my testbed and the relevant environment. Let’s see if this prevents new indexing failures from occuring on the Winlogbeat inputs. I’ll keep you posted.