Hi!
Whenever I import via Filebeat using the option “json.keys_under_root: true”, graylogs “Beats Input” ignores all custom fields and only imports the typical fields. This results in an empty message.
Other people were struggling with this too (https://discuss.elastic.co/t/sending-json-logs-from-filebeat-5-0-0-alpha5-1-to-graylog-2-2-0/64809 ) without understanding the actual source of the problem.
I checked the behaviour of filebeat - it’s as expected. The reason for the empty messages is graylogs “Beats Input”:
The input simply looks for message and some other special fields (like timestamp), ignoring all the rest. See BeatsCodec.parseFilebeat (https://github.com/Graylog2/graylog-plugin-beats/blob/master/src/main/java/org/graylog/plugins/beats/BeatsCodec.java).
I’m not saying this is wrong behaviour on the inputs part, but I don’t understand the purpose of this.
How I can effectively search and analyze logs from “beats input” without using an extractor to get important data into fields?
What am I missing here?
So far my only usable implementation was to not use “keys_under_root” and use a json-extractor afterwards. But thats everything but elegant and wastes lots of disk-space and resources (the message-field is not deletable afterwards).
Thanks in advance for your insights and help!
Bernhard
PS: just for clarification, what json.keys_under_root actually does in filebeat:
A message:
{ “a”:“abc”, “b”:“bcd”, “c”:“cde” }
usually results in:
{
"@timestamp": “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“message”: “{“a”:“abc”,“b”:“bcd”,“c”:“cde”}”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}
with keys_under_root the messages fields are put into the root-array:
{
"@timestamp": “…”,
“beat”: {
“hostname”: “…”,
“name”: “…”,
“version”: “5.5.2”
},
“a”: “abc”,
“b”: “bcd”,
“c”: “cde”,
“input_type”: “log”,
“offset”: …,
“source”: “…”,
“type”: “json”
}