GROK Troubles with end of logline

Hey guys,

I’m trying to create a set of GROK patterns to split firewall logs into their parts. To do so I’m combining multiple patterns like these:

FGFW_APPID appid=%{NONNEGINT:fgfw_appid}
FGFW_APPRISK apprisk=%{NOTSPACE:fgfw_apprisk}
FGFW_MSG msg=%{QS:fgfw_msg}

These are “bound” together with bigger pattern. The “important” thing is the optional ("*") white space at the end of the pattern which is used to divide the key/value patterns.


This one is then used to set up the full pattern that starts with a few static entries that are always there, while the ones used in the FGFW_KV_pattern above (which is used at the end) aren’t always in the same order or don’t exist all the time.

FGFW ^<%{NONNEGINT:fgfw_sylog}>date=%{TIMESTAMP_ISO8601_DATE:fgfw_date} time=%{TIME:fgfw_time} devname=%{NOTSPACE:fgfw_devname} devid=%{NOTSPACE:fgfw_devid} logid=%{NOTSPACE:fgfw_logid} type=%{NOTSPACE:fgfw_type} subtype=%{NOTSPACE:fgfw_subtype} %{FGFW_KV:fgfw_kv}

This works fine except for the last element which is always ignored for some reason. And I can’t wrap my head around why. The only idea I had was that it didn’t end with a whitespace, which is the reason why I made it “optional” with the *. I even tried moving the whitespace to the front of the pattern, but still the last element didn’t get captured.

Example log file entry:

<190>date=2018-09-21 time=09:38:41 devname=1112fw01 devid=FG1K5D3I15803336 logid=1059028704 type=utm subtype=app-ctrl eventtype=app-ctrl-all level=information vd=internet appid=16270 user="" srcip= srcport=123 srcintf="port35" dstip= dstport=123 dstintf="port36" profiletype="applist" proto=17 service="NTP" policyid=45 sessionid=320519228 applist="botnets" appcat="Network.Service" app="NTP" action=pass msg="Network.Service: NTP," apprisk=elevated

apprisk=elevated (and others if it ends with them) is not matched, even though it is included in the list above.

Any idea, what I’m doing wrong?

BTW: Is there a way to export all patterns? I already have accidentally deleted all patterns thanks to the “Replace all existing patterns?” feature which I interpreted as “Replace all existing patterns that are in the uploaded file?”

Heyo :slight_smile:

could this be a solution to your problem?


Hey Philipp,

I’ve tried adding the optional non-capturing group around different elements without results. But then, I was just poking around, not really sure where to put it in the first place. :-/

Ok, forget everything above. It was a PEBCAK problem after too much fiddling around with GROKs I guess.

I must have deleted the FGFW_APPRISK definition some time while fiddling around. So there was nothing there to match :-(.

While I’m pretty sure I tested that before, I just noticed that it didn’t get recognized either in a test message where it wasn’t the last element… I must’ve been blinded by too many ${}| before…

Ok, so now it works? :slight_smile:

At least it looks like it :wink: . I’m going to get some sleep tonight and check again tomorrow to make sure, I’m not dreaming or something…

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.