Winlogbeat templates for Elasticsearch not complete?

[Totally_Not_A_Robot (Tess)]

Wouldn’t you know it? Exactly two weeks after my previous thread, the same issue arises again. And I’m two hours too late to necro the previous thread.

Winlogbeat templates not included?

Hey guys, I’m mostly rubber-duckying here, apologies if the thread is mostly me answering my own questions… Situation Last week our Graylog cluster was puttering along just fine, everything fine and dandy. This morning, I come in to find no less than 38k indexer failures on the System > Overview. Troubleshooting The indexer failures details tell me that all of the errors revolve around one issue: type: mapper_parsing_exception, reason: failed to parse [winlogbeat_event_data_param2], cause …

This weekend the indices rotated and I’m back to looking at 18k indexing failures. And like before, it revolves around the “winlogbeat_event_data_param2” field, which is incorrectly typed in the new index.

I know. I was dumb. I should not have assumed that the Winlogbeat template for Elastic was actually complete. Turns out that the “winlogbeat.template.json” that I used (which came from the official Github) is missing fields like the aforementioned “winlogbeat_event_data_param2”. I’m not yet sure whether this particular field is custom to Graylog, whether the absence is intentional by the designers, or whether something else is up.

I’ll poke around and see what else I may have missed

EDIT 1 :
Found this old thread… could be useful.

In reply to that thread @jan wrote:

As long as you do not have the need to that data, I would drop it.

Now if only I had a passing notion of what he meant by that I have no clue how to tell Graylog or whomever it is that makes that field to stop making it

#EternalNewbie

EDIT 2:
Hmmmzzz, I guess that brings us to @jochen’s suggestion in this thread:

How to remove or disable particular field from the message in Graylog

Hi All, We are sending data from one graylog instance to another graylog instance using manage outputs. I have configured TCP Output in the stream and I given the destination graylog hostname in the corresponding field. In my destination graylog instance I have created new input GELF TCP with the port. I can see my data s are sending smoothly without any issues but when I checked in my destination graylog I can see the new field “forwarder” in my message and it contains the below value, forwa…

Something with pipelines and removing the field. Guess that could work, eh?

EDIT 3:
Okay, so it seems that what I want is to stop Graylog from splitting the “message” into “winlogbeat_event_data_param[1-30]”.

EDIT 4:
Okay! Found this thread on the ELK forums, from September 2016.

Event messages are splitted in event_data.param xy? How to fix that?

Basically it seems that I can’t easily prevent Winlogbeat from creating the fields. So either I add them all to the “winlogbeat.template.json”, or I use a Graylog pipeline to dike’m out…

EDIT 5:
If I go the route of the template for Elastic, it seems that I could get it sorted with the following:

"event_data_param1" : {
  "type" : "text",
  "include_in_all" : false,
  "index": no
},

Or leave of those false/no if you do want them indexed, but it really doesn’t serve any use.

[jan (Jan Doberstein)]

look at the data and act on that like I have mentioned in the edit1 linked thread.

Drop if no need for the data.

[Totally_Not_A_Robot (Tess)]

ACK… I’ll try and figure out how to do that with a pipeline. Thanks Jan.

[benvanstaveren]

remove_field or even drop_message are your friends

[Totally_Not_A_Robot (Tess)]

I swear, I’m totally not a robot…