Input Extractor Not Capture All Logs

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Currently have an input from OPNsense FW. The filterlog is parsing with the exception of the the following example log:

<134>Jul 29 12:26:15 gambit.*****.***** filterlog[40156]: 84,,,5ca69255279ee5173e47af3fb24bacef,bridge0,ip-option,pass,in,4,0xc0,,1,59085,0,DF,2,igmp,36,192.168.200.254,224.0.0.1,datalength=12

This is what it looks like not parsed:

image
image1800×480 47.8 KB

Other filterlogs such as the following parse without issue:

<134>Jul 29 12:26:15 gambit.*****.***** filterlog[40156]: 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,62,50557,0,DF,6,tcp,60,99.137.185.156,172.67.216.12,60624,443,0,S,2282301637,,64240,,mss;sackOK;TS;nop;wscale

Here is an example of what it looks like properly parsed:

image
image1803×849 53.6 KB

Here is the input extractor that I have currently:

{
  "extractors": [
    {
      "title": "OPNsense filterlog: IPv4 TCP",
      "extractor_type": "grok",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},tcp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{NUMBER:src_port},%{NUMBER:dst_port},%{NUMBER:data_length},%{DATA:tcp_flags},%{NUMBER:sequence_number},%{DATA:ack_number},%{NUMBER:window},%{DATA:urg},%{GREEDYDATA:tcp_options}",
        "named_captures_only": true
      },
      "condition_type": "string",
      "condition_value": "filterlog"
    },
    {
      "title": "OPNsense filterlog: IPv4 UDP",
      "extractor_type": "grok",
      "converters": [],
      "order": 1,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},udp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{NUMBER:src_port},%{NUMBER:dst_port},%{NUMBER:data_length}",
        "named_captures_only": true
      },
      "condition_type": "string",
      "condition_value": "filterlog"
    },
    {
      "title": "OPNsense filterlog: IPv4 ICMP",
      "extractor_type": "grok",
      "converters": [],
      "order": 2,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},icmp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{GREEDYDATA:icmp_type}",
        "named_captures_only": true
      },
      "condition_type": "string",
      "condition_value": "filterlog"
    },
    {
      "title": "OPNsense filterlog: IPv4 Other Protocols",
      "extractor_type": "grok",
      "converters": [],
      "order": 3,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},%{WORD:protocol},%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{GREEDYDATA:additional_info}",
        "named_captures_only": true
      },
      "condition_type": "string",
      "condition_value": "filterlog"
    },
    {
      "title": "OPNsense Suricata",
      "extractor_type": "grok",
      "converters": [],
      "order": 4,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} suricata\\[%{NUMBER:pid}\\]: %{DATA:info} \\[Priority: %{NUMBER:priority}\\] \\{%{DATA:protocol}\\} %{IP:src_ip}:%{NUMBER:src_port} -> %{IP:dst_ip}:%{NUMBER:dst_port}",
        "named_captures_only": true
      },
      "condition_type": "string",
      "condition_value": "suricata"
    },
    {
      "title": "SourceIP geo-lookup-country",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 5,
      "cursor_strategy": "copy",
      "source_field": "src_ip",
      "target_field": "src_country",
      "extractor_config": {
        "lookup_table_name": "geo-lookup-country"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "DestIP geo-lookup-country",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 6,
      "cursor_strategy": "copy",
      "source_field": "dst_ip",
      "target_field": "dst_country",
      "extractor_config": {
        "lookup_table_name": "geo-lookup-country"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "5.0.0"
}

I am looking for some Ideas on what either I am doing incorrectly or a way to improve the extractor.

2. Describe your environment:

  • OS Information:
    CPU: AMD Ryzen 9 6900HX (16) @ 3.30 GHz
    GPU: AMD Rembrandt @ 2.40 GHz [Integrated]
    Memory: 12.24 GiB / 28.15 GiB (43%)
    SHELL: bash 5.1.16
    OS: Ubuntu jammy 22.04 x86_64
    KERNEL: Linux 5.15.0-112-generic

  • Package Version:

graylog-6.0-repository/stable,now 1-1 all [installed]
graylog-server/stable,now 6.0.4-1 amd64 [installed]
libmongoc-1.0-0/jammy,now 1.21.0-1build1 amd64 [installed,automatic]
libmongocrypt0/jammy,now 1.3.0-1ubuntu1 amd64 [installed,automatic]
mongodb-database-tools/jammy,now 100.9.5 amd64 [installed,upgradable to: 100.10.0]
mongodb-mongosh/jammy,now 1.9.0 amd64 [installed,upgradable to: 2.2.14]
mongodb-org-database-tools-extra/jammy,now 6.0.16 amd64 [installed,automatic]
mongodb-org-database/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16]
mongodb-org-mongos/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16]
mongodb-org-server/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16]
mongodb-org-shell/jammy,now 6.0.16 amd64 [installed,automatic]
mongodb-org-tools/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16]
mongodb-org/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16]
opensearch/stable,now 2.15.0 amd64 [installed]
  • Service logs, configurations, and environment variables:

n/a

3. What steps have you already taken to try and solve the problem?

I have tried adjusting the extractor with no success…

4. How can the community help?

I am looking for some Ideas on what either I am doing incorrectly or a way to improve the extractor.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.