Grok in pipeline

[samkh]

Hi,

i have configured a pipeline that attached to a stream. in the pipeline is a stage0 that has one rule.
the rule is:
rule “format logs”
when
true
then
let message = to_string($message.message);
let fw = grok(pattern: “%{SYSLOGTIMESTAMP} %{GREEDYDATA} fw=”%{DATA:source}" pri=%{INT:priority} msg="%{DATA:identifer}: %{DATA:loglevel}: %{DATA:message}"", value: “message”, only_named_captures: true);
set_fields(fw);
end

i did change the order in the system configuration that the message filter chain before the pipeline processor.
but the message still be not extracted with the grok function.
when i use the pattern in as a grok extractor, it works fine.
does anybody has a hint for me?

[tmacgbay]

A couple of tips I use to help with GROK:

Use the debug() function to figure out what is going on in your pipeline rule - maybe something like this:

debug(concat("=== This is fw: ", to_string(fw)));

and watch for it in your Graylog logs with:

tail -f /var/log/graylog-server/server.log

Also helpful, create your GROK under system ->GROK patterns and just reference it - makes for a cleaner rule.

Lastly, practice your grok against the message at this site.

[samkh]

thank you for your answer.
i got in the logs the following line:
2021-11-29T08:42:59.233+01:00 INFO [Function] PIPELINE DEBUG: === This is fw: {}
i created a grok pattern, but it is still not working.
also practice the grok pattern against a message as a extractor in the graylog clutser and it works as a extractor.

[tmacgbay]

Please repost the rule using the forum formatting tools - specifically </> As it looks above, you have too many double quotes in the grok statement but it’s hard to tell without being properly formatted. There should only be double quotes at the beginning and end of the pattern: section any others need to be escaped if you are looking for them. Also it would help if you posted a sample message to test against.

[samkh]

rule "format mellanox"
when
  true
then
  let message = to_string($message.message);
  debug(concat("=== This is message: ", to_string(message)));
  let fw = grok(pattern: "%{SYSLOGTIMESTAMP} %{GREEDYDATA} fw=\"%{DATA:source}\" pri=%{INT:priority} msg=\"%{DATA:identifer}: %{DATA:loglevel}: %{DATA:message}\"", value: "message", only_named_captures: true);
  set_fields(fw);

  debug(concat("=== This is fw: ", to_string(fw)));
end

that was the rule.
An example of a message:

Nov 29 13:49:47 id=firewall time="2021-11-29 13:49:47" fw="fqdn.example" pri=5 msg="snmpd[8925]: [snmpd.NOTICE]: Got SNMP request from ip x.x.x.x"

[tmacgbay]

@samkh

let message = to_string($message.message);

This can be confusing when you call your variable ‘message’ use something like ‘msg’ or ‘the_thing’ or something like that.

let the_thing = to_string($message.message);

(personal pref)

@samkh

...%{DATA:message}\""....

This part of your GROK is where I think it is erroring out (and relevant to my first point) - your message already has a message field… best to name that something else.

[samkh]

i have done the changes but the problem still there.

[tmacgbay]

Does that include moving your GROK to grok patterns and referencing it? I had forgotten - any escaped character done in pipeline has to be double escaped \\" … which you don’t have to do if you move the full GROK to the GROK Patterns section and reference it in your pipeline.

[samkh]

i move the full Grok to Grok Pattern und i edit the value in the grok function to “to_string(the_thing)” after that it works fine.