What is the most effective way to parse logs with lots of optionnal fields?

[FrenchToast]

Good morning, I am completely new to Graylog (or any SIEM really) and I was tasked to parse our firewall’s logs.

There are around 40 total fields with only 10 being there all the time so the 30+ others are optionnal.

I have been using one pipeline and GROK to parse the logs so I have a lot of ? because of all the optionnal fields.

Is this the way to do it or is there a more efficient way?

Thank you,

[gsmith]

Hello && Welcome @FrenchToast

Easiest way I now of is using a different INPUT Raw/plaintext UDP/TCP this should limit you number of fields and then create the fields needed either with pipeline or extractor.

[tmacgbay]

Here is a note on how I handled firewall traffic coming from PaloAlto firewalls:

Unable to Import Palo Alto Networks Content Packs

I tried to be good and create a content pack but man that interface is just soooo wonky… All I was trying to do is put out pipelines and their rules and extract the extra stuff I have in there… AND you have to get it 100% right because although I can edit all I want in github, once you post in Graylog Marketplace, the only thing you can change is the title (LAME!) Ok… done complaining. I am posting my rules in sequence here. I am assuming your input works and you know how to …

[gsmith]

I just read that post , Thankfully we don’t use PaloAlto . @tmacgbay someone on discord was asking about dashboard with PA.

[tmacgbay]

@gsmith

dashboard with PA

hmmm … maybe post mine for the contest…