K=V extractor help

[syntax]

I’m trying to use pipeline rules to dynamically extract the key/pair values.

I took reference from here,

Key Value Parser Delivers Useful Information Fast | Graylog

My log looks something like this,

2021-04-19 10:45:34 Message details here, key1: value1, key2: value2
2021-04-19 10:45:34 ## Another format of message details here, key1: value1, key2: value2

Here is what I’m trying to do…
my issue is extracting the timestamp and the message details.

rule "key_value_parser"
when
    has_field("message")
then
    set_fields(key_value(
            value: to_string($message.message), 
            delimiters: ",",
            kv_delimiters: ":",
            trim_value_chars: "",
            trim_key_chars:"",
    ));
end

by the way, how does $message.message come about?

[shoothub]

HI @syntax

1. Why you try to use delimeter ; if your message uses , delimeter?
2. Best way would be first to extract KV part from message using regex or GROK. Please post some real word examples so we can help.

[syntax]

hi @shoothub,

1. thanks. fixed the error.
2. extractor runs before pipeline right? in that case, i can use grok pattern extractor followed by pipeline. does my logic sound right?

[shoothub]

2. Order of processing extractors and pipeline depends on configuration on Message Processors Configuration in System - Configuration. If using extractor, check if Pipeline processor is below Message Filter Chain, if you want to use extracted field in pipeline rules.

https://docs.graylog.org/en/4.0/pages/pipelines/stream_connections.html#the-importance-of-message-processor-ordering

Also note, that you can use grok() pipeline function in same (or new) pipeline rule as key_value pipeline function is you want, it’s not necessary to use extractor at all. It depends on what do you prefer.
https://docs.graylog.org/en/4.0/pages/pipelines/functions.html#grok

[syntax]

thank you for the reference links.
i know how to do it now.