Hello,
I’m trying to replicate a Beat input behavior for tags with Pipelines and “flatten_json”
In Filebeat I declare a file like this with tags :
- type: log
paths:
- "/var/log/haproxy.log"
tags: ["haproxy", "access"]
In Graylog with a Beat input,
Then when I search a message i have a field like this :
filebeat_tags
["haproxy","access"]
When I click on the value and I choose “Add to query”, this is what I see in the search field :
filebeat_tags:haproxy,access
Which is great, because in can search one of the value, or several by adding more AND filebeat_tag:<tag>
In Graylog with a Kafka RAW input,
This is the JSON eaten by Graylog :
{"@timestamp":"2023-02-06T15:15:00.410Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.6.1"},"log":{"offset":2965630,"file":{"path":"/var/log/haproxy.log"}},"message":"Feb 6 16:14:59 localhost haproxy[81865]: 178.237.98.45:65088 [06/Feb/2023:16:14:59.941] www~ graylogs/pirv-siem-es-graylog-03 0/0/1/9/11 200 620 - - ---- 4/4/1/0/0 0/0 \"GET /api/system/cluster/nodes HTTP/1.1\"","tags":["haproxy","access"],"input":{"type":"log"},"host":{"name":"pirv-siem-lb.services.com"},"agent":{"ephemeral_id":"34e1af49-e08c-4c64-a6aa-a25db055d715","id":"06fba243-d033-450e-8c43-e9e55cb43669","name":"pirv-siem-lb.services.com","type":"filebeat","version":"8.6.1"},"ecs":{"version":"8.0.0"}}
My Pipeline to convert it :
rule "Simple Flatten Json Rule"
when
true
then
let sJson = to_string($message.message);
let fJson = flatten_json(value: sJson, array_handler: "json", stringify: false);
set_fields(to_map(fJson));
end
Then when I search a message I see a field for my tags , who looked the same as the other :
tags
["haproxy","access"]
BUT When I click on the value and I choose “Add to query”, this is what I see in the search field :
tags:\[\"haproxy\",\"access\"\]
The behavior is not the same as the input BEAT, now I have a lot of "" and it’s not really usable.
Any idea why this is happening ?
or How to get a array of value from my JSON to get a usable tag list ?
Thanks.