Key=value extractor and quotes / remove whitespaces with regex and replace

(helias) #1


I’m looking for a suggestion to decode message which come in form of key=ip-adress key=value key=timestamp key=“value”:

time=15:24:30 devname=NAME-1 devid=SD230N3G17966 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip= srcport=55230 srcintf="wan1" dstip= dstport=445 dstintf="wan1" poluuid=ef0f1e78-84c6-99a739745c6b sessionid=81335928 proto=6 action=deny policyid=3 policytype=policy dstcountry="Germany" srccountry="Brazil" trandisp=noop service="SMB" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel=high

I tried to extract it using the key=value extractor as suggested in the manual. But it seems like this is not working with the " special character.

So I thought i just remove all the " from the message via an regex and replace extractor and afterwards extract the values via the k=v extractor.

But now I’m stuck on what to enter in the “replace” field. Just entering nothing (what I want in the end) is not working.

Is this even the right way to do it?
Any help is much appreciated!

(Jochen) #2

What does that mean exactly?

(helias) #3

I never got it to work with the above message. Ok, I think I might be wrong. This was the error

{"type":"mapper_parsing_exception","reason":"failed to parse [level]","caused_by":{"type":"number_format_exception","reason":"For input string: \"notice\""}}

I don’t know why I thought it was the ".
But nevertheless it did not work. Can you still help?

(Jochen) #4

The “level” field is usually a numeric value (see in Graylog.

You could use the processing pipeline to parse the key/value pairs ( and then rename the “level” field to something else (

(helias) #5

Ok thanks for the answer!
If I understand the pipelines correctly, I would have to define a pipline-key-value-extractor for every key in the message, right?

For a quick-win couldn’t I just regex-replace “level” (I don’t really need the field) with another string and then run the k=v extractor?

(Jochen) #6

No, why would you? The key_value() provides basically the same functionality as the key/value converter.

(helias) #7

Ok thanks again.
I think I have to do some more reading.
I changed level to lvl via regexp-replace and afterwards I run the k=v extractor. Works as expected but I will check out the pipeline key_value() functionality.

Thanks again for the quick help!

(Jan Doberstein) #8

a processing rule would look like:

rule "kv_ex_prefix"
    // extract all key-value from "message" and prefix it with kv_ 
                            value: to_string($message.message), 
                            trim_value_chars: "\""
                prefix: "kv_"


This would also strip out the " if present.

(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.