Pipeline Rule not working - Source replace with grok

Graylog Central (peer support)
1

Hello, I am trying to do a simple replace of the source from where I get syslogs as its currently not correct. I have been playing with this for quite some time but I cant get it to work. If I use this same grok command on an extractor I get exactly what I want but not in the rule.

Sorry very new…

rule "FQDN to Source"
when
  has_field("message")
then
 let extract = grok(pattern: "\\s%{HOSTNAME}", value: to_string($message.source));
 set_fields(extract);
end
2

What does $message.source look like - it helps to know what you are starting from… you may have to escape more… you could try "\\\\s%{HOSTNAME}"

3

Thanks, that did not seem to make a difference:

Here is the message

xxxx:xx:xx:xx::x swc1.command.net: 657001 Base SYSTEM-WARNING-tmnxConfigDelete-2008 [OAM]:  Test name "GlobalCommandPing-211932", owner name "TiMOS CLI", ping managed object deleted

The Source is showing up like this:

xxxx:xx:xx:xx::x

I want the source to be “swc1.command.net”

4

You can avoid playing with double escapes and such by creating the combined GROK in system->GROK patterns and just calling that.

on a side note - GROK can get expensive unless you tie it down - for instance, the one you have will examine the entire message for the pattern…which individually doesn’t matter but if you start processing hundreds or thousands of those it might. "^/S %{HOSTNAME}" Would tie it to the beginning of the message (with the ^) and stop if it didn’t immediately work.

5

Thank you, I tried both your suggestions and the source is still showing up as the IPV6 address rather than the hostname. I feel like I’m doing something fundamentally wrong. My goal it to search the message and then replace the source with what I found in the message.

1st Recommendation:
GROK Pattern “FQDN” = “\s%{HOSTNAME}”

rule "FQDN to Source"
when
  has_field("message")
then
 let extract = grok(pattern: "%{FQDN}", value: to_string($message.source));
 set_fields(extract);
end

2nd recommendation:

rule "FQDN to Source"
when
  has_field("message")
then
 let extract = grok(pattern: "^/S %{HOSTNAME}", value: to_string($message.source));
 set_fields(extract);
end
6

Hmm… in the GROK statement you are only evaluating $message.source which would only be the source field. You are trying to grok the hostname from the entire message so the line should be:

let extract = grok(  pattern: "%{WORD} %{HOSTNAME:hostname}", value: to_string($message.message), only_named_captures: true;
set_fields(extract);

OK, I modified it a bit more than that but it eliminates the escaped characters

7

Using what you posted does not seem to make any difference, here is a full example of what I get in the graylog search field when I open a message.

HOSTNAME
swc1.command.net

facility
local6

facility_num
22

full_message
<180>Jul  8 20:08:47 xxxx:xx:xx:xx::x swc1.command.net: 658711 Base SYSTEM-WARNING-tmnxConfigModify-2006 [OAM]:  Test name "GlobalCommandPing-212492", owner name "TiMOS CLI", ICMP ping configuration modified

level
4

message
xxxx:xx:xx:xx::x swc1.command.net: 658711 Base SYSTEM-WARNING-tmnxConfigModify-2006 [OAM]:  Test name "GlobalCommandPing-212492", owner name "TiMOS CLI", ICMP ping configuration modified

source
xxxx:xx:xx:xx::x

timestamp
2021-07-08 20:08:47.000 +00:00

Also as a test I changed my rule to this and I get the expected result:

rule "FQDN to Source"
when
  has_field("message")
then
let extract = grok(  pattern: "%{WORD} %{HOSTNAME:hostname}", value: to_string($message.message), only_named_captures: true);
set_fields(extract);
set_field("source","TESTING");
end

Source in the logs are now “TESTING”

8

If you already have HOSTNAME broken out, couldn’t you simply put in

set_field("source", to_string($message.HOSTNAME));

?

9

Yes that looks to be working most of the time!!! It might take some time to propagate or I might have to restart some things. But on most devices they are showing the hostname as the source.

Thank you so much for your help, you helped me learn a lot will mark the solution soon to see if everything is working as expected.

rule "FQDN to Source"
when
  has_field("message")
then
  set_field("source", to_string($message.HOSTNAME));
end
10

IT was a bit roundabout but it looks as though we got there

1 Like
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.