Unifi USG-3P firewall rules and GROK pattern

[schneich1 (Schneich)]

Dear community,

It has been almost 2 years, since a last did some work on my Graylog. It is running in a docker container and I keep it up-to-date. (currently on version 6.1.3)

I get the syslog messages from my Unifi USG-3P into graylog for a long time.
No, I would like to create a dashboard, which provides an overview on incoming traffic.

This is an example of a message:

USG-3P kernel: [LAN_IN-2025-A]IN=eth1.20 OUT=eth1.40 MAC=24:5a:4c:7b:37:aa:6a:ba:b5:5c:10:a5:08:00:45:00:00:3c SRC=192.168.20.3 DST=192.168.40.40 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=22257 DF PROTO=TCP SPT=39452 DPT=55123 WINDOW=65535 RES=0x00 SYN URGP=0

I came up with this GROK pattern, which should work fine, except for the part, where I try to get the firewall rule out of the “[” “]”.

^%{DATA:device_name} kernel: \\[%{DATA:firewall_rule}\\]IN=%{DATA:interface_in} OUT=%{DATA:interface_out} MAC=%{MAC:source_mac}:%{MAC:destination_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:source_ip} DST=%{IP:destination_ip}.*?PROTO=%{WORD:network_transport} SPT=%{NUMBER:source_port} DPT=%{NUMBER:destination_port} .*?$

I have tried to use this pattern in a pipeline rule like this:

rule "Unifi USG-3P firewall messages incoming traffic"
when
  contains(to_string($message.source), "usg", true)
then
  set_fields(
    grok(
      pattern:"^%{DATA:device_name} kernel: \\[%{DATA:firewall_rule}\\]IN=%{DATA:interface_in} OUT=%{DATA:interface_out} MAC=%{MAC:source_mac}:%{MAC:destination_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:source_ip} DST=%{IP:destination_ip}.*?PROTO=%{WORD:network_transport} SPT=%{NUMBER:source_port} DPT=%{NUMBER:destination_port} .*?$",
      value:to_string($message.message),
      only_named_captures: true
    )
  );
end

In regex, the “[” “]” have a special meaning. I tried to escape them, but that did not work. So how do I do that?

[Wine_Merchant]

Hey @schneich1

The double escape should be enough and the below is parsing the message when simulating the rule in my environment.

rule "Unifi USG-3P firewall messages incoming traffic"
when
true
then
  set_fields(
    grok(
      pattern:"%{DATA:host_name} kernel: \\[%{DATA:firewall_rule}\\]IN=%{DATA:interface_in} OUT=%{DATA:interface_out} MAC=%{MAC:source_mac}:%{MAC:destination_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:source_ip} DST=%{IP:destination_ip}.*?PROTO=%{WORD:network_transport} SPT=%{NUMBER:source_port} DPT=%{NUMBER:destination_port}.*?$",
      value:to_string($message.message),
      only_named_captures: true
    )
  );
end

[schneich1 (Schneich)]

Hi @Wine_Merchant,

thank you for testing. I just have realized that there are several versions of firewall messages provided by the Unifi Network Application.

After making sure that I test the right message with the right Grok Pattern, it is working for me as well.

[schneich1 (Schneich)]

Hi @Wine_Merchant,

please allow me a - probably very simple - follow up question:
So there are different versions of firewall messages containing different fields. I can easily create a second Grok Pattern to capture the fields from the second version of messages.
for example:

USG-3P kernel: [WAN_LOCAL-default-D]IN=eth0 OUT= MAC=01:00:5e:00:00:01:54:67:51:c3:0e:75:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=2

Where do I put the second Grok Pattern? Can I add a second stage in my Pipeline or do I have to include this in my first Pipeline Rule?
(The underlying question is: if a messages is processed through Stage 0 in a pipeline, will it be handed on to Stage 1 or will it be omitted from the Pipeline?)

[schneich1 (Schneich)]

I just have realised that I can add two Grok Patterns to one Stage. Is this the way to do it?

I made sure that a message can only be true for one of the Grok Patterns, as I filter with a “starts_with”.

Rule 1

rule "Unifi USG-3P firewall parse [WAN_LOCAL-default-D]"
when
  starts_with(
  value: to_string($message."message"),
  prefix: "USG-3P kernel: [WAN_LOCAL-default-D]"
)
then
  set_fields(
    grok(
      pattern:"^%{DATA:device_name} kernel: \\[%{DATA:firewall_rule}\\]IN=%{GREEDYDATA:interface_in} .* MAC=%{MAC:destination_mac}:%{MAC:source_mac}:%{GREEDYDATA:ethernet_header} .*?SRC=%{IP:source_ip} DST=%{IP:destination_ip}.*? PROTO=%{WORD:network_transport}.*?$",
      value:to_string($message.message),
      only_named_captures: true
    )
  );
end

Rule 2

rule "Unifi USG-3P firewall parse [LAN_IN]"
when
  starts_with(
  value: to_string($message."message"),
  prefix: "USG-3P kernel: [LAN_IN"
)
then
  set_fields(
    grok(
      pattern:"^%{DATA:device_name} kernel: \\[%{DATA:firewall_rule}\\]IN=%{GREEDYDATA:interface_in} OUT=%{GREEDYDATA:interface_out} MAC=%{MAC:destination_mac}:%{MAC:source_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:source_ip} DST=%{IP:destination_ip}.*? PROTO=%{WORD:network_transport} SPT=%{NUMBER:source_port} DPT=%{NUMBER:destination_port} .*?$",
      value:to_string($message.message),
      only_named_captures: true
    )
  );
end

[Wine_Merchant]

Hello @schneich1,

That would be one way, there is now a multi grok function (as of 6.1 maybe) which could also be useful to you. Example can be found here.

[schneich1 (Schneich)]

There have been some more types of Log messages - I would have to create 5 or 6 Grok Pattern, all very similar with minor differences. Therefore, I have decided to build a more flexible pattern that captures all the message variations.

It took me some time, but I figured out, how to use (?: ... ) and combine it with ? to get some optional values or fields.

rule "Unifi USG-3P firewall parse messages"
when
  starts_with(
  value: to_string($message."message"),
  prefix: "USG-3P kernel"
)
then
  set_fields(
    grok(
      pattern:"^%{DATA:device_name} %{WORD:application}: \\[%{DATA:firewall_rule}\\]IN=%{DATA:interface_in} OUT=(?:%{DATA:interface_out})? MAC=%{MAC:destination_mac}:%{MAC:source_mac}:%{GREEDYDATA:ethernet_header} SRC=%{IP:source_ip} DST=%{IP:destination_ip} .*?PROTO=%{WORD:network_transport}(?: SPT=%{NUMBER:source_port})?(?: DPT=%{NUMBER:destination_port})?.*?$",
      value:to_string($message.message),
      only_named_captures: true
    )
  );
end

Has this a big impact on the need of resources for my little docker container?

[schneich1 (Schneich)]

I start to like the Graylog functionality

@Wine_Merchant Thank you for your help