Messages do not get through a pipeline - How to debug?

[lukas.pavljuk]

Hello,

I had to modify an input from TCP Syslog to UDP Syslog (As one of our apps we want to use to send messages from into Graylog does not support TCP Syslog), but after removing the old input, creating a new one and connecting a stream onto the input, despite not changing anything else, messages no longer get correctly processed by a subsequent pipeline.

I can see messages getting correctly assigned into the custom stream, can use the simulator to see the pipeline would correctly process the messages and create all the important fields… Yet, in a dashboard that worked before, I get almost no messages. Most messages the dashboard loads do not have any of the fields the pipeline was supposed to extract.

The pipeline in use can be found here , it is a pipeline taken directly from a guide in the Graylog Github repo here, only the regex is fixed. Same with the stream definition.

Is there some setting somewhere I didn’t notice? I’d be less confused if even the pipeline simulator failed to parse and process the messages, but it doesn’t! It correctly parses the messages and assigns the new fields to correct values. And on top of all that, some messages still get parsed right. Like…

This message parsed correctly:

[1:384:5] ICMP PING [Classification: Misc activity] [Priority: 3] {ICMP} *remote-ip* -> *local-ip*</169>

The next one, however, didn’t parse:

[1:384:5] ICMP PING [Classification: Misc activity] [Priority: 3] {ICMP} *remote-ip* -> *local-ip*</169>

((Just to provide a complete picture, the full messages before getting parsed and stripped in the custom stream I have are here:

<169>0 2021-06-09T15:22:33.942228+02:00 *hostname* snort 15157 - -  [1:384:5] ICMP PING [Classification: Misc activity] [Priority: 3] {ICMP} *remote-ip* -> *local-ip*</169>
<169>0 2021-06-09T15:27:34.837870+02:00 *hostname* snort 15157 - -  [1:384:5] ICMP PING [Classification: Misc activity] [Priority: 3] {ICMP} *remote-ip* -> *local-ip*</169>

The delta between these two messages was about 5 minutes, which suggests I probably set something somewhere for a moment, which allowed the messages to parse correctly. However, I cannot find what that setting could have been.

Graylog in use - v4.0.7
Platform: Debian 10 Linux

The server is bound onto a LAN IP, with an Nginx reverse-proxy used to access the UI over plaintext HTTP.

[tmacgbay]

You can use the debug function in your pipeline to see what it is doing - the link has an example on how to add it…
https://docs.graylog.org/en/latest/pages/pipelines/functions.html#debug

The results will show up in your Graylog log:
tail -f /var/log/graylog-server/server.log

You can post the code in the forum if you like. Use the </> tool to make the formatting nice. My work has pastebin blocked. (grrr)

[lukas.pavljuk]

Okay, I used the debug function, and considering I get no debug messages in the log, something is probably very wrong with the pipeline.

This is my current pipeline stage 0 rule.

rule "Extract Snort alert fields"
when
  has_field("message")
then
  debug("Processing message in snort pipeline - ".$message.message);
  let m = regex("^\\s?\\[(\\d+):(\\d+):(\\d+)\\] (.+?) \\[Classification: (.+?)\\] \\[Priority: (\\d+)] \\{(.+?)\\} ((?:\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})|(?:(?:[0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,7}:|(?:[0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,5}(?::[0-9a-fA-F]{1,4}){1,2}|(?:[0-9a-fA-F]{1,4}:){1,4}(?::[0-9a-fA-F]{1,4}){1,3}|(?:[0-9a-fA-F]{1,4}:){1,3}(?::[0-9a-fA-F]{1,4}){1,4}|(?:[0-9a-fA-F]{1,4}:){1,2}(?::[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:(?:(?::[0-9a-fA-F]{1,4}){1,6})|:(?:(?::[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(?::[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(?:ffff(?::0{1,4}){0,1}:){0,1}(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])|(?:[0-9a-fA-F]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])))(:(\\d{1,5}))? -> ((?:\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})|(?:(?:[0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,7}:|(?:[0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|(?:[0-9a-fA-F]{1,4}:){1,5}(?::[0-9a-fA-F]{1,4}){1,2}|(?:[0-9a-fA-F]{1,4}:){1,4}(?::[0-9a-fA-F]{1,4}){1,3}|(?:[0-9a-fA-F]{1,4}:){1,3}(?::[0-9a-fA-F]{1,4}){1,4}|(?:[0-9a-fA-F]{1,4}:){1,2}(?::[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:(?:(?::[0-9a-fA-F]{1,4}){1,6})|:(?:(?::[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(?::[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(?:ffff(?::0{1,4}){0,1}:){0,1}(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])|(?:[0-9a-fA-F]{1,4}:){1,4}:(?:(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(?:25[0-5]|(?:2[0-4]|1{0,1}[0-9]){0,1}[0-9])))(:(\\d{1,5}))?\\R?", to_string($message.message));

  set_field("snort_alert", true);

  set_field("generator_id", m["0"]);
  set_field("signature_id", m["1"]);
  set_field("signature_revision_id", m["2"]);

  set_field("description", m["3"]);
  set_field("classification", m["4"]);
  set_field("priority", to_long(m["5"]));
  set_field("protocol", m["6"]);

  set_field("src_addr", m["7"]);
  set_field("src_port", to_long(m["9"]));

  set_field("dst_addr", m["10"]);
  set_field("dst_port", to_long(m["12"]));
end

The pipeline is connected to a Snort stream, which takes messages from the All Messages stream, matches them against these two rules:

message must match regular expression ^\s?\[\d+:\d+:\d+].*
application_name must match exactly snort

And when I use the stream tester, messages do pass both of the rules and get correctly routed into the stream correctly. The issue therefore has to be somewhere in the pipeline.

[shoothub]

I don’t think your usage of debug function is correct, please try rather this one:

debug(concat("Processing message in snort pipeline - ", to_string($message.message)));

[lukas.pavljuk]

Okay, thank you, I modified the debug statement (A few hours back already), and still got no messages in the logfile, hinting at a possible pipeline issue, as I made sure I did get messages into the appropriate stream the pipeline’s supposed to be connected to.

[localhost (MFL)]

Hi @lukas.pavljuk,
how do you route your messages to the stream? If you use stream rules, can you check the order of your message processors?

I suspect it might have something to do with your issue, if you have the Pipeline Processor in front of the Message Filter Chain, your pipeline rules will never be triggered (messages are only routed to the stream after the pipeline handling is finished.) See Stream connections — Graylog 4.0.0 documentation for more background.

Cheers,

l0calhost

[lukas.pavljuk]

Okay, yes, it was indeed an issue with the message processor order. Putting the pipeline processor as last fixed it. Thank you so much!

[localhost (MFL)]

Great to hear it works now. My pleasure.