Hi!
Whenever I import via Filebeat using the option “json.keys_under_root: true”, graylogs “Beats Input” ignores all custom fields and only imports the typical fields. This results in an empty message.
Other people were struggling with this too (Sending Json Logs from Filebeat 5.0.0_alpha5-1 to Graylog 2.2.0 - Beats - Discuss the Elastic Stack ) without understanding the actual source of the problem.
I checked the behaviour of filebeat - it’s as expected. The reason for the empty messages is graylogs “Beats Input”:
The input simply looks for message and some other special fields (like timestamp), ignoring all the rest. See BeatsCodec.parseFilebeat (graylog-plugin-beats/src/main/java/org/graylog/plugins/beats/BeatsCodec.java at master · Graylog2/graylog-plugin-beats · GitHub).
I’m not saying this is wrong behaviour on the inputs part, but I don’t understand the purpose of this.
How I can effectively search and analyze logs from “beats input” without using an extractor to get important data into fields?
What am I missing here?
So far my only usable implementation was to not use “keys_under_root” and use a json-extractor afterwards. But thats everything but elegant and wastes lots of disk-space and resources (the message-field is not deletable afterwards).
Thanks in advance for your insights and help!
Bernhard
PS: just for clarification, what json.keys_under_root actually does in filebeat:
A message:
{ “a”:“abc”, “b”:“bcd”, “c”:“cde” }
usually results in:
{
“@timestamp”: “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“message”: “{"a":"abc","b":"bcd","c":"cde"}”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}
with keys_under_root the messages fields are put into the root-array:
{
“@timestamp”: “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“a”: “abc”,
“b”: “bcd”,
“c”: “cde”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}