Unable to Implement a Simple Pipeline Rule to Manipulate the source Field to Reflect the Hostname in the message Field

[kwakalack (James)]

Hello community,

I’m having some issues with pipeline rules and hoping I can find some assistance please.

My stage 0 rule is working fine and I have it set to continue processing on next stage when none or more rules on this stage match. The stage 0 rule simply states when there is a message field that contains “cad:” in it, set the source field as gl2_remote_ip.

rule “Siklu”
when
has_field(“message”) &&
contains(to_string($message.message), “cad:”)
then
set_field(“source”, to_string($message.gl2_remote_ip));
end

For my stage 1 rule, I’m attempting to use the hostname string in the message field (for logs from devices of a particular vendor) as the source field. If you take a look at this screenshot, Graylog grabs the very first part of the message and uses it as the source as it assumes it’s the hostname.

The 3 numbers in the front of the message and hostname (in this case wtm4800-18l-80hi-2ft-2) are unique from log to log.

Here’s what I’ve tried so far where I used regex101.com to verify the regex formulas match the hostnames in the messages (without the leading or ending spaces) and I also included the extra escape "" required for Graylog’s pipeline syntax:

1. When logs have a message field that begins with 3 digits followed by a space, then, split the message into parts using space as boundaries and use the second part (part 1) as the source field.

rule “Aviat”
when
has_field(“message”) &&
regex(“^\d{3}\s”, to_string($message.message)).matches == true
then
let parts = split(" ", to_string($message.message));
set_field(“source”, parts[1]);
end

2. When logs have a message field that contains “wtm” in it, then match everything from left to right using space as boundaries and use it as the source field.

rule “Aviat1”
when
has_field(“message”) &&
contains(to_string($message.message), “wtm”)
then
let matches = regex(“\b[^\s]wtm[^\s]\b”, to_string($message.message));
set_field(“source”, matches[“0”]);
end

3. When logs have a message field with 3 digits follow by a space, then remove it from the message (tried to force the hostname to be the leading part of the message).

rule “Aviat1”
when
has_field(“message”) &&
regex(“^\d{3}\s”, to_string($message.message)).matches == true
then
let cleaned = regex_replace(“^\d{3}\s+”, “”, to_string($message.message));
set_field(“message”, cleaned);
end

4. I even tried going super simple by just saying if a log has a message field, split it into parts and use the second part as source field.

rule “Aviat1”
when
has_field(“message”)
then
let parts = split(" ", to_string($message.message));
set_field(“part1_candidate”, parts[1]);
end

None of these worked. Methods 1 and 3 would be most ideal as it won’t trigger any potential false positives but I’m hoping I’m just running into some formula/syntax issues and someone may be able to spot the discrepancy.

Any help is appreciated and thank you very much in advance!

[kwakalack (James)]

Ps: It seems as those the extra backslashes (extra escapes for regex) have been omitted in the post even though I’ve included them.

[Joel_Duffield]

Ya my guess would be some regex escapes or something are missing, have you seen this post Need some help with regex in pipeline rule - #2 by drewmiranda-gl

[drewmiranda-gl (Drew Miranda)]

Are you able to use the code function here to paste the raw text?

You can enclose your “code” (e.g. Pipeline Rule) by adding ``` above and below your text. Example:

```
this is code
```

which will render as.

this is code

This helps us greatly to rule out issues caused by text or formatting issues with this forum. Thanks!

At a quick glance one thing i noticed is you are referencing the result of the regex using a int type (number only) however, the graylog regex result is only accessible using a string, even if that string is a number. So instead of parts[1] you should do parts["1"]. Its also a good idea to be explicit about the data types you save into fields using the to_ functions. In this example, set_field(“part1_candidate”, parts[1]); would look like:

set_field(“part1_candidate”, to_string(parts["1"]));

This should be covered in the post that @Joel_Duffield linked to above. Give that a read through and see if that doesn’t answer your questions and solve your issues. If you still have questions/issues let us know!