Graylog 4.X Not Parsing All JSON Fields with parse_json()

[jandrusk (Justin Andrusk)]

I’m sending Suricata logs from PFSense to Syslog-NG to Graylog. I have a Graylog Pipeline with the following rule that parses some, but not all of the embedded JSON fields:

rule “suricata_strip”
when contains(to_string($message.message), “kiera suricata”) then
let m = regex_replace(“kiera suricata: “, to_string($message.message),to_string(””) );
let json_fields = parse_json(m);
set_fields(to_map(json_fields));
end

I’m using a regex_replace to remove the leading string that Syslog-NG is adding, so it’s stripped down to pure JSON. However, when I looked at one of the messages only certain fields are parsed into new fields. Attaching two screenshots of a sample of a partially parsed message. Some messages are not parsed at all.

Any ideas?

[shoothub]

Try to use json extractor, sometimes extractor is better in parsing than pipeline rule function.