Graylog Update to 6.0.5 (Graylog 6.0.5+3ef5be7) No longer able to use existing Grok Patterns

secdoc · August 8, 2024, 12:30pm

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Upgraded Graylog Open from 6.0.4 to 6.0.5. After Doing so, the Grok patterns that were previously working within the Input Extractors and the Pipeline Rules no longer function.

2. Describe your environment:

OS Information:

CPU:  AMD Ryzen 9 6900HX (16) @ 3.30 GHz
GPU:  AMD Rembrandt @ 2.40 GHz [Integrated]
Disk (/) 854.17 GiB / 1.83 TiB (46%) - ext4
Memory:  12.94 GiB / 28.15 GiB (46%)
SWAP:  2.53 GiB / 8.00 GiB (32%)
SHELL:  bash 5.1.16
OS:  Ubuntu jammy 22.04 x86_64
KERNEL:  Linux 5.15.0-117-generic
PACKAGES:  755 (dpkg), 3 (snap)

Package Version:

dpkg -l | grep -E ".(elasticsearch|graylog|mongo)."
ic  graylog-5.0-repository                 1-2                                     all          Package to install Graylog 5.0 GPG key and repository
ic  graylog-5.1-repository                 1-2                                     all          Package to install Graylog 5.1 GPG key and repository
ic  graylog-5.2-repository                 1-2                                     all          Package to install Graylog 5.2 GPG key and repository
ii  graylog-6.0-repository                 1-1                                     all          Package to install Graylog 6.0 GPG key and repository
hc  graylog-enterprise                     5.1.0-6                                 amd64        Graylog Enterprise Server
ii  graylog-server                         6.0.5-1                                 amd64        Graylog server
ii  libmongoc-1.0-0                        1.21.0-1build1                          amd64        MongoDB C client library - runtime files
ii  libmongocrypt0:amd64                   1.3.0-1ubuntu1                          amd64        client-side field level encryption library - runtime files
ii  mongodb-database-tools                 100.10.0                                amd64        mongodb-database-tools package provides tools for working with the MongoDB server: 
hi  mongodb-mongosh                        1.9.0                                   amd64        MongoDB Shell CLI REPL Package
hi  mongodb-org                            6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
hi  mongodb-org-database                   6.0.6                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database-tools-extra       6.0.16                                  amd64        Extra MongoDB database tools
hi  mongodb-org-mongos                     6.0.6                                   amd64        MongoDB sharded cluster query router
hi  mongodb-org-server                     6.0.6                                   amd64        MongoDB database server
ii  mongodb-org-shell                      6.0.16                                  amd64        MongoDB shell client
hi  mongodb-org-tools                      6.0.6                                   amd64        MongoDB tools

Service logs, configurations, and environment variables:

The config and setup is fairly vanilla

3. What steps have you already taken to try and solve the problem?

I have tried rewriting grok to see if there is anything that might change the results but either get error shown in screenshot above or no successful parse of the log.

This is what is showing after the update:

and there is sample logs that should parse properly but do not:

<134>1 2024-08-08T12:50:10+00:00 gambit.acme.com filterlog 9057 - [meta sequenceId="70186"] 59,,,13144bee09802860dc782932e41b0603,bridge0,match,block,in,4,0x0,,245,54321,0,none,6,tcp,44,35.203.210.25,10.13.37.0,54525,5172,0,S,2131746833,,65535,,mss

<134>1 2024-08-08T12:50:07+00:00 gambit.acme.com filterlog 9057 - [meta sequenceId="70185"] 90,,,ac73efefebe804c4ea27776e7803bd8f,igc0,match,pass,out,4,0x0,,64,58617,0,none,17,udp,135,192.168.88.2,192.168.2.152,5546,1516,115

<134>1 2024-08-08T12:50:07+00:00 gambit.acme.com filterlog 9057 - [meta sequenceId="70182"] 90,,,ac73efefebe804c4ea27776e7803bd8f,igc0,match,pass,out,4,0x0,,64,6224,0,none,17,udp,134,192.168.88.2,192.168.2.152,6391,1516,114

<134>1 2024-08-08T12:50:06+00:00 gambit.acme.com filterlog 9057 - [meta sequenceId="70168"] 71,,,77f0efdab49017defd129875497465fb,bridge0,match,pass,in,4,0x0,,64,26005,0,DF,6,tcp,60,10.13.37.0,3.33.157.109,39934,443,0,S,361717497,,42340,,mss;sackOK;TS;nop;wscale

<134>1 2024-08-08T12:50:05+00:00 gambit.acme.com filterlog 9057 - [meta sequenceId="70145"] 88,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,63,29138,0,none,17,udp,81,10.13.37.0,202.12.31.53,61150,53,61

Here are example grok patterns:

OPNsense filter log:

<%{INT:syslog_priority}>%{INT} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} filterlog %{INT:pid} - \[meta sequenceId="%{INT:sequence_id}"\] %{INT:rule_number},%{DATA:sub_rule_number},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},match,%{WORD:action},%{WORD:direction},%{INT:ip_version},%{DATA:tos},%{DATA:ecn},%{INT:ttl},%{INT:id},%{INT:offset},%{DATA:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IPV4:src_ip},%{IPV4:dst_ip},%{INT:src_port},%{INT:dst_port}(,%{INT:data_length}(,%{DATA:tcp_flags}(,%{GREEDYDATA:tcp_options})?)?)?

OPNsense Suricata log:

<%{POSINT:syslog_priority}>%{INT} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} suricata %{INT:pid} - \[meta sequenceId="%{INT:sequence_id}"\] \[%{INT:signature_id}:%{INT:signature_rev}:%{INT:signature_update}\] %{DATA:alert_message} \[Classification: %{DATA:classification}\] \[Priority: %{INT:priority}\] \{%{WORD:protocol}\} %{IPV4:src_ip}:%{INT:src_port} -> %{IPV4:dst_ip}:%{INT:dst_port}

4. How can the community help?

I am trying to understand what changed and how to correct. As previously stated, these patterns were functional and were working well, but now the data is no longer getting parsed which affects dashboards and so on…

Any assistance in how to resolve would be greatly appreciated.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

drewmiranda-gl · August 8, 2024, 4:46pm

I took a quick look in my test environment. I started with 6.0.4 and used your sample message and grok and validated the extractor matched as expected. I then updated to 6.0.5 and repeated the test. I did find that the extractor is still matching:

Unfortunately I was not able to replicate the issue as you described. It appears the message ‘We were not able to run the grok extraction. Please check your parameters.’ only indicates the grok pattern didn’t match sample message provided.

Can you clarify if your existing GROK extractors no longer work after upgrading Graylog 6.0.4 to 6.0.5 or is the issue the extractor editor page isn’t properly matching in the way you expect?

Can you provide the message (even if you need to partially redact it) that your screenshot above is failing to match?

secdoc · August 8, 2024, 5:03pm

@drewmiranda-gl Thank you…The examples are exactly what exactly I was using for testing and were working before the update because I wrote them to update the parsing after the update to OPNsense 24.7, which resulted in a change in the log output format, when I initially setup the extractor, that being said, as I was doing additional troubleshooting, I ended up creating a pipeline rule using the same grok pattern (see below) and then it started parsing. Prior to that point it was only through the input extractor.

rule "parse_opnsense_247_filterlog"
when
    has_field("full_message")
then
    let parsed = grok(
        "<%{INT:syslog_priority}>%{INT} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} filterlog %{INT:pid} - \\[meta sequenceId=\"%{INT:sequence_id}\"\\] %{INT:rule_number},%{DATA:sub_rule_number},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},match,%{WORD:action},%{WORD:direction},%{INT:ip_version},%{DATA:tos},%{DATA:ecn},%{INT:ttl},%{INT:id},%{INT:offset},%{DATA:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IPV4:src_ip},%{IPV4:dst_ip},%{INT:src_port},%{INT:dst_port}(,%{INT:data_length}(,%{DATA:tcp_flags}(,%{GREEDYDATA:tcp_options})?)?)?",
        to_string($message.full_message)
    );

    set_fields(parsed);

    // Convert numeric fields to appropriate types
    set_field("syslog_priority", to_long(parsed.syslog_priority));
    set_field("pid", to_long(parsed.pid));
    set_field("sequence_id", to_long(parsed.sequence_id));
    set_field("rule_number", to_long(parsed.rule_number));
    set_field("ip_version", to_long(parsed.ip_version));
    set_field("ttl", to_long(parsed.ttl));
    set_field("id", to_long(parsed.id));
    set_field("offset", to_long(parsed.offset));
    set_field("protocol_id", to_long(parsed.protocol_id));
    set_field("length", to_long(parsed.length));
    set_field("src_port", to_long(parsed.src_port));
    set_field("dst_port", to_long(parsed.dst_port));
    set_field("data_length", to_long(parsed.data_length));

    // Add some derived fields
    set_field("log_type", "opnsense_filterlog");
    
    // Create a summary field
    set_field("summary", 
        "OPNsense Filterlog: " + 
        to_string(parsed.action) + " " +
        to_string(parsed.direction) + " " +
        to_string(parsed.protocol) + " traffic from " +
        to_string(parsed.src_ip) + ":" + to_string(parsed.src_port) +
        " to " + to_string(parsed.dst_ip) + ":" + to_string(parsed.dst_port)
    );
end

drewmiranda-gl · August 8, 2024, 6:01pm

Good to hear the pipeline is working! All things being equal the pipeline will generally perform better than the extractor (in terms of CPU usage/message throughput). Plus pipelines offer a much richer feature set and greater flexibility.

Have you tested the new “set field type” or “field type profiles” It should let you define explicit field mappings that would negate the need to re set fields and add type hints type to_long.

secdoc · August 8, 2024, 6:43pm

@drewmmiranda Thank you. I have not tried the new capabilities. Is there docs on the capabilities and/or mappings?

drewmiranda-gl · August 8, 2024, 7:17pm

This doc is a good start: Field Types

let me know if you have any questions or get stuck anywhere!

system · August 22, 2024, 7:17pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Made the jump from Graylog 4 to 5 ? Please share your feedback! Graylog Central (peer support)	8	882	February 23, 2023
Cannot upgrade graylog open to 6.0.7 Graylog Central (peer support)	9	111	November 11, 2024
Download Graylog V5.0 now and post your impressions, questions, and suggestions Graylog Central (peer support)	21	4254	January 6, 2023
Graylog 4.3 to 5.1 not start Graylog Central (peer support)	5	497	October 15, 2023
Help with upgrading for a graylog beginner Graylog Central (peer support)	7	1661	December 31, 2021

Graylog Update to 6.0.5 (Graylog 6.0.5+3ef5be7) No longer able to use existing Grok Patterns

Related topics