Graylog pipeline keys with spaces

[syntax]

Does anyone know if graylog pipeline rule supports keys with spaces? Mine seems to completely skip the parsing.

set_fields (
    key_value(
        value: $message.message,
        delimiters: ",",
        kv_delimiters: ":",
        trim_value_chars: "",
        trim_key_chars:""
    )
);

e.g. test key:value

[shoothub]

Hi @syntax

I don’t think so. Please post your real example messages, so we can help. Maybe replace space with something will work.

[syntax]

@shoothub

example:

2021-06-19 14:23:15 Message details, file: c:\users\admin\desktop\file.exe, log type: alert, ip address: 127.0.0.1

i have about a thousand log formats…so adding a dash/underscore would be very tedious.

[tmacgbay]

There is already a feature request out there to solve this that has been sitting around for a bit (since Jan 2020)- here is the original post I had on it that includes a link to the feature request to handle it:

Key_value - remove space in field portion

I have a key_value result but the fields portion have spaces that I need to convert to underscore to use the set_fields() function. I can use regex_replace -but- the regex I have matches on spaces in the key and value portion. So for instance: client os version=microsoft windows 10 pro 64-bit becomes client_os_version=microsoft_windows_10_pro_64-bit when what I want is: client_os_version=microsoft windows 10 pro 64-bit The usable but not optimal regex with example data (https://regex101…

Get all your friends to look at it and comment that it should be prioritized…

[syntax]

@tmacgbay interesting. how did u manage to circumvent the issue?

[tmacgbay]

I had a previous rule that broke out the event_description field as defined by a quoted section of the message, then broke out the portion that needed key_value() applied, cleaning up the spaces and commas. Note how the regex is non-capturing for the first two words, then once it’s done its work I am referencing indexes. Also note the commented out debug() functions so I could watch what this looked like as it went through. Not pretty and likely not efficient, but I didn’t have thousands of these coming in (small company)

    let e_message     = to_string($message.event_description);

    let desc_parts      = regex(pattern: "^(?:\\w+\\s+){2}(.*)\\.\\s+(.+)", value: e_message);
    set_field("event_action", to_string(desc_parts["0"]));
    let desc_lowered    = replace(lowercase(to_string(desc_parts["1"]))," , ", ", ");    //might have extranious comma's
    let desc_cleaned    = regex_replace("\\b\\s+", desc_lowered , "_");                 //replace unwonted spaces 
    let keyed_up        = key_value(desc_cleaned,
                                ",",
                                ":",
                                true,
                                true,
                                "take_last",
                                " ",
                                " "
                    );

    //debug("$$$$---Event to be :");
    //debug(to_string(keyed_up));

    set_fields(keyed_up);