Receiving messages are stopping when running set_fields in pipeline

[Nklimko (Mikalai Klimko)]

Hi!
I have some troubles with using grok patterns in Graylog pipelines.
I have a simple test rule to parse message field:

rule "pipeline"
when
    true
then
    set_field("pipeline","true");
    let mess = to_string($message.message);
    let patt = grok(pattern:"%{GREEDYDATA:message2}",value: mess,only_named_captures: true);
    set_fields(patt);
end

It works correctly in Simulation, but when I including this rule into stage all messages are stopping to receive in Graylog.
When I just commented string set_fields(patt);, messages are going on correctly receiving.
It repeats with any trying to post grok results in field: set_field(“message2”,patt[“message2”]); etc.
But set_field(“pipeline”,“true”); - setting additional field with hardcoded params is working correctly.

Where is the problem?

Graylog Enterprise 4.2 (docker)

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[tmacgbay]

It appears the way you have your Grok set up, it will grab the entire message and put it into field message2 This is because your GROK: %{GREEDYDATA:message2} says to do exactly that.

You can use an online GROK Debugger to figure what your GROK would do.

For instance if you had the following message:

Commit job started processing. Dequeue time=2022/01/26 08:03:40. JobId=168.User: Bob. Commit Description: Remove DMZ NAT - RTZ34 plus tags

and you applied this GROK pattern:

(?:.*User:)%{SPACE}%{WORD:fw_admin}(?:.*Description:)%{GREEDYDATA:thing_that_happened}

You would get this result:

{
  "fw_admin": [
    [
      "Bob"
    ]
  ],
  "thing_that_happened": [
    [
      " Remove DMZ NAT - RTZ34 plus tags"
    ]
  ]
}

Play around with GROK a bit and see if you can capture your message, if you need help, be sure to post a sample obfuscated message and the GROK you have progressed to…