Pipeline rule to split log messages into key-value fields not working

Graylog Central (peer support)
, ,
3

Check to see if if your Message Filter Chain is before your Pipeline Processor under Message Processor Configuration, IF NOT - this often causes no results in the pipeline. Also, you can watch in the logs to see what is going on using the debug() function - I added it below with a note that shows where to look for the debug message. here are some possible results:

  • You see the full message debug showing up in the logs but the field are not split out with key_value() — this means there is some error in the key_value() I have seen key_value() fail silently without logs or anything just because there was a key it didn’t like (had spaces)… the portion subj==unconfined key="mdatp" looks odd and may cause a failure… something to consider.

  • If you are NOT seeing the debug messages in the log, the when statement might be having issues… change it to just true to catch all things passing by the rule

  • If you have true in your when and it is still not seeing the full messages, then it is possible the pipeline/rule is not connected to the stream that has the message coming in.

rule "auditd_keys_to_fields"
when
    starts_with ((to_string($message.message)) , "type", true )
then
     // use $ tail -f /var/log/graylog-server/server.log to watch for the results of the below debug message
    debug(concat("++++++++++ Full Message: ", to_string($message.message))); 

    // extract all key-value from "message" and prefix it with auditd_
    set_fields(
        fields: key_value(
                    value: to_string($message.message),
                    trim_value_chars: "\""
                ),
        prefix: "auditd_"
        );
end