Why doesn't Beats Input support keys_under_root?


(Bernhard Wege) #1

Hi!

Whenever I import via Filebeat using the option “json.keys_under_root: true”, graylogs “Beats Input” ignores all custom fields and only imports the typical fields. This results in an empty message.
Other people were struggling with this too (https://discuss.elastic.co/t/sending-json-logs-from-filebeat-5-0-0-alpha5-1-to-graylog-2-2-0/64809 ) without understanding the actual source of the problem.

I checked the behaviour of filebeat - it’s as expected. The reason for the empty messages is graylogs “Beats Input”:
The input simply looks for message and some other special fields (like timestamp), ignoring all the rest. See BeatsCodec.parseFilebeat (https://github.com/Graylog2/graylog-plugin-beats/blob/master/src/main/java/org/graylog/plugins/beats/BeatsCodec.java).

I’m not saying this is wrong behaviour on the inputs part, but I don’t understand the purpose of this.
How I can effectively search and analyze logs from “beats input” without using an extractor to get important data into fields?

What am I missing here?

So far my only usable implementation was to not use “keys_under_root” and use a json-extractor afterwards. But thats everything but elegant and wastes lots of disk-space and resources (the message-field is not deletable afterwards).

Thanks in advance for your insights and help!
Bernhard

PS: just for clarification, what json.keys_under_root actually does in filebeat:

A message:

{ “a”:“abc”, “b”:“bcd”, “c”:“cde” }

usually results in:

{
"@timestamp": “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“message”: “{“a”:“abc”,“b”:“bcd”,“c”:“cde”}”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}

with keys_under_root the messages fields are put into the root-array:

{
"@timestamp": “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“a”: “abc”,
“b”: “bcd”,
“c”: “cde”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}


(system) #2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.