GROK Troubles with end of logline

Graylog Central (peer support)
1

Hey guys,

I’m trying to create a set of GROK patterns to split firewall logs into their parts. To do so I’m combining multiple patterns like these:

FGFW_APPID appid=%{NONNEGINT:fgfw_appid}
FGFW_APPRISK apprisk=%{NOTSPACE:fgfw_apprisk}
FGFW_MSG msg=%{QS:fgfw_msg}
...

These are “bound” together with bigger pattern. The “important” thing is the optional ("*") white space at the end of the pattern which is used to divide the key/value patterns.

FGFW_KV ((%{FGFW_ACTION}|%{FGFW_APP}|%{FGFW_APPCAT}|%{FGFW_APPID}|%{FGFW_APPLIST}|%{FGFW_CRACTION}|%{FGFW_CRLEVEL}|%{FGFW_CRSCORE}|%{FGFW_DSTCOUNTRY}|%{FGFW_DSTIP}|%{FGFW_DSTPORT}|%{FGFW_DSTINTF}|%{FGFW_DURATION}|%{FGFW_EVENTTYPE}|%{FGFW_HOSTNAME}|%{FGFW_LEVEL}|%{FGFW_MSG}|%{FGFW_POLICYID}|%{FGFW_POLICYTYPE}|%{FGFW_POLUUID}|%{FGFW_PROFILETYPE}|%{FGFW_PROTO}|%{FGFW_RCVDBYTE}|%{FGFW_SENTBYTE}|%{FGFW_SENTPKT}|%{FGFW_SERVICE}|%{FGFW_SESSIONID}|%{FGFW_SRCCOUNTRY}|%{FGFW_SRCIP}|%{FGFW_SRCPORT}|%{FGFW_SRCINTF}|%{FGFW_TRANDISP}|%{FGFW_URL}|%{FGFW_USER}|%{FGFW_VD})\s*)*

This one is then used to set up the full pattern that starts with a few static entries that are always there, while the ones used in the FGFW_KV_pattern above (which is used at the end) aren’t always in the same order or don’t exist all the time.

FGFW ^<%{NONNEGINT:fgfw_sylog}>date=%{TIMESTAMP_ISO8601_DATE:fgfw_date} time=%{TIME:fgfw_time} devname=%{NOTSPACE:fgfw_devname} devid=%{NOTSPACE:fgfw_devid} logid=%{NOTSPACE:fgfw_logid} type=%{NOTSPACE:fgfw_type} subtype=%{NOTSPACE:fgfw_subtype} %{FGFW_KV:fgfw_kv}

This works fine except for the last element which is always ignored for some reason. And I can’t wrap my head around why. The only idea I had was that it didn’t end with a whitespace, which is the reason why I made it “optional” with the *. I even tried moving the whitespace to the front of the pattern, but still the last element didn’t get captured.

Example log file entry:

<190>date=2018-09-21 time=09:38:41 devname=1112fw01 devid=FG1K5D3I15803336 logid=1059028704 type=utm subtype=app-ctrl eventtype=app-ctrl-all level=information vd=internet appid=16270 user="" srcip=10.42.10.129 srcport=123 srcintf="port35" dstip=46.165.221.137 dstport=123 dstintf="port36" profiletype="applist" proto=17 service="NTP" policyid=45 sessionid=320519228 applist="botnets" appcat="Network.Service" app="NTP" action=pass msg="Network.Service: NTP," apprisk=elevated

apprisk=elevated (and others if it ends with them) is not matched, even though it is included in the list above.

Any idea, what I’m doing wrong?

BTW: Is there a way to export all patterns? I already have accidentally deleted all patterns thanks to the “Replace all existing patterns?” feature which I interpreted as “Replace all existing patterns that are in the uploaded file?”

2

Heyo :slight_smile:

could this be a solution to your problem?

Greetings,
Philipp

3

Hey Philipp,

I’ve tried adding the optional non-capturing group around different elements without results. But then, I was just poking around, not really sure where to put it in the first place. :-/

4

Ok, forget everything above. It was a PEBCAK problem after too much fiddling around with GROKs I guess.

I must have deleted the FGFW_APPRISK definition some time while fiddling around. So there was nothing there to match :-(.

While I’m pretty sure I tested that before, I just noticed that it didn’t get recognized either in a test message where it wasn’t the last element… I must’ve been blinded by too many ${}| before…

5

Ok, so now it works? :slight_smile:

6

At least it looks like it :wink: . I’m going to get some sleep tonight and check again tomorrow to make sure, I’m not dreaming or something…

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.