Pipeline rule does not save and no errors

Graylog Central (peer support)
1

1. Describe your incident:

Creating a pipeline rule that parses dnsmasq blacklisted requests based on the following sample logs:

<13>Nov 29 18:43:53 silversurfer dnsmasq[335109]: regex blacklisted analytics.twingate.com is 0.0.0.0
<13>Nov 29 18:43:53 silversurfer dnsmasq[335109]: regex blacklisted analytics.twingate.com is ::

The following rule is what is placed within the source code editor:

rule "parse_dnsmasq_blacklist"
when
    has_field("message")
then
    let message = to_string($message.message);

    // Regular expression to parse the log entry
    let regex = "<(\\d+)>([\\w\\s:]+)\\s(\\S+)\\s+dnsmasq\\[(\\d+)\\]:\\s+regex\\sblacklisted\\s(\\S+)\\sis\\s(\\S+)";
    
    // Apply the regex to the message
    let match = regex(regex, message);

    // Set fields based on regex matches
    set_field("syslog_priority", to_long(match["0"]));
    set_field("timestamp", match["1"]);
    set_field("hostname", match["2"]);
    set_field("process_id", to_long(match["3"]));
    set_field("blacklisted_domain", match["4"]);
    set_field("blacklist_response", match["5"]);
end

When running a test validation or trying to save the rule, nothing happens and there are no errors as typically seen within the editor if there is a problem.

2. Describe your environment:

  • OS Information:
Graylog Graylog 5.1.0+14ba491
OS: Ubuntu 22.04.2 LTS x86_64
Kernel: 5.15.0-72-generic
Shell: bash 5.1.16
CPU: AMD Ryzen 9 6900HX with Radeon Graphics (16) @ 3.300GHz
Memory: 4978MiB / 28839MiB
  • Package Version:
secdoc@cerebro:~$ dpkg -l | grep -E ".(opensearch|graylog|mongo)"
ic  graylog-5.0-repository                1-2                                     all          Package to install Graylog 5.0 GPG key and repository
ic  graylog-5.1-repository                1-2                                     all          Package to install Graylog 5.1 GPG key and repository
ii  graylog-5.2-repository                1-2                                     all          Package to install Graylog 5.2 GPG key and repository
hc  graylog-enterprise                    5.1.0-6                                 amd64        Graylog Enterprise Server
hi  graylog-server                        5.2.0-7                                 amd64        Graylog server
ii  libmongoc-1.0-0                       1.21.0-1build1                          amd64        MongoDB C client library - runtime files
ii  libmongocrypt0:amd64                  1.3.0-1ubuntu1                          amd64        client-side field level encryption library - runtime files
ii  mongodb-database-tools                100.9.1                                 amd64        mongodb-database-tools package provides tools for working with the MongoDB server: 
hi  mongodb-mongosh                       1.9.0                                   amd64        MongoDB Shell CLI REPL Package
hi  mongodb-org                           6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
hi  mongodb-org-database                  6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database-tools-extra      6.0.11                                  amd64        Extra MongoDB database tools
hi  mongodb-org-mongos                    6.0.6                                   amd64        MongoDB sharded cluster query router
hi  mongodb-org-server                    6.0.6                                   amd64        MongoDB database server
ii  mongodb-org-shell                     6.0.11                                  amd64        MongoDB shell client
hi  mongodb-org-tools                     6.0.6                                   amd64        MongoDB tools
ii  opensearch                            2.11.0                                  amd64        An open source distributed and RESTful search engine
  • Service logs, configurations, and environment variables:

n/a

3. What steps have you already taken to try and solve the problem?
I have tried modifying other rules and making changes or updates with a different set of information. I am able to change and update other rules.

4. How can the community help?

Trying to understand why the rule will not save. Any ideas or insight would be helpful.

2

This is a very old (and embarrassing) issue: you cannot call a variable match.

I will try to raise priority on this.

Meanwhile: when running into puzzling rule issues, the easiest debugging approach is to comment out statements until you identify the offending line. Then you can focus on that. That’s how I found this problem and the corresponding issue.

3

Incidentally, I located the entire list of reserved words in our ANTLR grammar. Don’t use any of these as variable names, regardless of casing.

All
Either
Pass
And
Or
Not
Pipeline
Rule
During
Stage
When
Then
End
Let
Match

4

@patrickmann Thank you for the insight. I went through and renamed the variable as follows:

when
    has_field("message")
then
    let message = to_string($message.message);

    // Regular expression to parse the log entry
    let regex = "<(\\d+)>([\\w\\s:]+)\\s(\\S+)\\s+dnsmasq\\[(\\d+)\\]:\\s+regex\\sblacklisted\\s(\\S+)\\sis\\s(\\S+)";
    
    // Apply the regex to the message
    let blaclist_match = regex(regex, message);

    // Set fields based on regex matches
    set_field("syslog_priority", to_long(blaclist_match["0"]));
    set_field("timestamp", blaclist_match["1"]);
    set_field("hostname", blaclist_match["2"]);
    set_field("process_id", to_long(blaclist_match["3"]));
    set_field("blacklisted_domain", blaclist_match["4"]);
    set_field("blacklist_response", blaclist_match["5"]);
end

It now accepts the rule, but still not seeing the fields being broken out. I tried with an if statement as well, but the rule editor does not like the if…

rule "parse_dnsmasq_blacklist_log"
when
    has_field("message")
then
    let message = to_string($message.message);

    // Simplified regular expression to parse the log entry
    let regex = "<(\\d+)>([\\w\\s:]+)\\s(\\S+)\\s+dnsmasq\\[(\\d+)\\]:\\s+regex\\sblacklisted\\s(\\S+)\\sis\\s(\\S+)";
    
    // Apply the regex to the message
    let blacklist_match = regex(regex, message);

    // Set fields if regex blacklist_matches
    if (blacklist_match != null) {
        set_field("syslog_priority", to_long(blacklist_match["0"]));
        set_field("timestamp", blacklist_match["1"]);
        set_field("hostname", blacklist_match["2"]);
        set_field("process_id", to_long(blacklist_match["3"]));
        set_field("blacklisted_domain", blacklist_match["4"]);
        set_field("blacklist_response", blacklist_match["5"]);
    }
end

Any thoughts on what I may be doing wrong?

5

You don’t need to escape the escape character "".

Are you sure your regex capture groups are correct? I tried throwing it into regex101 and it didn’t match your sample data.

6

@patrickmann thank you for the feedback. I was trying to create a parser that could be handled with a single regex, but that is a no go, so I broke it up…

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.