Key=value extractor and quotes / remove whitespaces with regex and replace

Graylog Central (peer support)
1

Hi

I’m looking for a suggestion to decode message which come in form of key=ip-adress key=value key=timestamp key=“value”:

time=15:24:30 devname=NAME-1 devid=SD230N3G17966 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=001.002.003.004 srcport=55230 srcintf="wan1" dstip=008.009.010.011 dstport=445 dstintf="wan1" poluuid=ef0f1e78-84c6-99a739745c6b sessionid=81335928 proto=6 action=deny policyid=3 policytype=policy dstcountry="Germany" srccountry="Brazil" trandisp=noop service="SMB" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel=high

I tried to extract it using the key=value extractor as suggested in the manual. But it seems like this is not working with the " special character.

So I thought i just remove all the " from the message via an regex and replace extractor and afterwards extract the values via the k=v extractor.

But now I’m stuck on what to enter in the “replace” field. Just entering nothing (what I want in the end) is not working.

Is this even the right way to do it?
Any help is much appreciated!

2

What does that mean exactly?

3

I never got it to work with the above message. Ok, I think I might be wrong. This was the error

{"type":"mapper_parsing_exception","reason":"failed to parse [level]","caused_by":{"type":"number_format_exception","reason":"For input string: \"notice\""}}

I don’t know why I thought it was the ".
But nevertheless it did not work. Can you still help?

4

The “level” field is usually a numeric value (see https://en.wikipedia.org/wiki/Syslog#Severity_level) in Graylog.

You could use the processing pipeline to parse the key/value pairs (http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#key-value) and then rename the “level” field to something else (http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#rename-field).

5

Ok thanks for the answer!
If I understand the pipelines correctly, I would have to define a pipline-key-value-extractor for every key in the message, right?

For a quick-win couldn’t I just regex-replace “level” (I don’t really need the field) with another string and then run the k=v extractor?

6

No, why would you? The key_value() provides basically the same functionality as the key/value converter.

7

Ok thanks again.
I think I have to do some more reading.
I changed level to lvl via regexp-replace and afterwards I run the k=v extractor. Works as expected but I will check out the pipeline key_value() functionality.

Thanks again for the quick help!

8

a processing rule would look like:

rule "kv_ex_prefix"
when
    has_field("kv")
then
    // extract all key-value from "message" and prefix it with kv_ 
    set_fields(
                fields: 
                        key_value(
                            value: to_string($message.message), 
                            trim_value_chars: "\""
                            ),
                prefix: "kv_"
            );

end

This would also strip out the " if present.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.