Grok Pattern in Pipelines

[idscomm]

Hello folks,

I am new to Graylog, I started configuring my server, adding stream from my UniFi Gateway (which works perfectly). I am now trying to improve the message log by applying pipelines (since I read not to use extractor anymore). Issue is that I managed to get a Grok pattern working with the extractor but it is not working with my pipeline rule and I don’t know why… Although I am a Network Admin, this is more on the programming side which has never been my forte… Can anyone help me getting started so I can wrap my head around this pipelines function?

This is the raw message:
Gateway [LAN_OUT-RET-20001] DESCR=“VPN - Allowed Servers” IN=br0 OUT=br1 MAC= SRC=192.168.1.100 DST=192.168.2.1 LEN=40 TOS=00 PREC=0x00 TTL=127 ID=9921 DF PROTO=TCP SPT=53629 DPT=80 SEQ=2234031635 ACK=581856961 WINDOW=6142 ACK URGP=0 MARK=1a0000

This is my working Grok Pattern:
Gateway [%{DATA:Direction}] DESCR=“%{DATA:Rule_Description}” IN=%{DATA:From_Network} OUT=%{DATA:To_Network} MAC=%{DATA:mac} SRC=%{IP:Source_IP} DST=%{IP:Destination_IP} LEN=%{INT:length} TOS=%{DATA:tos} PREC=%{DATA:prec} TTL=%{INT:ttl} ID=%{INT:id} %{DATA:flags} PROTO=%{DATA:Protocol} SPT=%{INT:Source_Port} DPT=%{INT:Destination_Port} SEQ=%{INT:seq} ACK=%{INT:ack} WINDOW=%{INT:window} %{DATA:ack_flags} URGP=%{INT:urgp} MARK=%{DATA:mark}

I tried using the Grok simulator, AI, etc. but I am literally stuck…

Thanks in advance!

** Running Graylog 6.0.7+4779d72 on graylog (Eclipse Adoptium 17.0.12 on Linux 5.15.0-122-generic)*

[Joel_Duffield]

I wouldnt use grok for that as its overkill, i would use some regex to remove the first two values, or grab them if you need them, and then i would use the key value function for the rest, it should make quick work of that and use much less cpu. And its alot easier to write.

[idscomm]

Thanks for the reply. The only data in this raw message that I am looking for are:

[LAN_OUT-RET-20001]
DESCR=“VPN - Allowed Servers”
IN=br0
OUT=br1
SRC=192.168.1.100
DST=192.168.2.1
PROTO=TCP
SPT=53629
DPT=80

How would I proceed to build my pipeline rule in this case?

[Wine_Merchant]

hey @idscomm

Use the below to split into fields and then use either remove_single_field or remove_multiple_fields function to drop what is not required.

rule "Key Value"
when
true
then

set_fields(
   fields:key_value(
   value: to_string($message.message),
   delimiters:" ",
   kv_delimiters:"="
)
);

end

[idscomm]

Thank you for your answer, I will give it a shot later today. Appreciate the help!