Pipeline Rule: Issues with key_value function (fields can be created in Simulator, but not in real time)

[bettels-uhi (Constantin Bettels)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
When using the rule simulator, the fields get set, when a log is being processed, the fields don’t get created, even though it’s the same message text.
2. Describe your environment:

• OS Information:
  Ubuntu 20.04
• Package Version:
  Graylog 5.2.10
• Service logs, configurations, and environment variables:
  Here is an example of a log file that should be processed:

2024-09-05 09:14:13,807: username=yddasdasda, status=Failed Login, ipAddress=192.168.2.140, entryPoint=Universal API

Here is my pipeline rule:

rule "Jamf access log"
when
    has_field("message") &&
    $message.source == "jamf" 
then
    let messageText = to_string($message.message);
    debug(concat("Original message: ", messageText));
    
    let pattern = "%{TIMESTAMP_ISO8601:timestamp}:%{GREEDYDATA}";
    let value_array = grok(pattern, messageText);
    let timestamp = value_array["timestamp"];
    let timestamp_string = concat(to_string(timestamp), ":");
    set_field("jamf_access_log_timestamp", timestamp);
    
    debug(concat("Extracted timestamp: ", to_string(timestamp)));
    
    let message_for_key_value_map = regex_replace("\\Q" + timestamp_string + "\\E",messageText,"",false);
    //debug(concat("Message after removing timestamp: ", message_for_key_value_map));
    
    let key_value_pairs = key_value(
        value: to_string(message_for_key_value_map),
        trim_key_chars: " ",
        trim_value_chars: " ",
        delimiters: ",",
        kv_delimiters: ":"
    );
    
    debug(concat("Key-value pairs: ", to_string(key_value_pairs)));
    
    let entryPoint = key_value_pairs["entryPoint"];
    let ipAddress = key_value_pairs["ipAddress"];
    let jamf_status = key_value_pairs["status"];
    let username = key_value_pairs["username"];
    
    debug(concat("EntryPoint: ", to_string(entryPoint)));
    debug(concat("IPAddress: ", to_string(ipAddress)));
    debug(concat("Status: ", to_string(jamf_status)));
    debug(concat("Username: ", to_string(username)));
    
    set_field("jamf_entryPoint", entryPoint);
    set_field("jamf_username", username);
    set_field("jamf_ip_address", ipAddress);
    
    let geo = lookup("geoip", to_string(ipAddress)); 
    set_field("jamf_ip_address_geolocation", geo["country"].iso_code);
    
    set_field("jamf_access_log_status", jamf_status);
end

When trying the rule in the pipeline rule simulator, the logs look like this:
2024-09-05T12:19:48.665+02:00 INFO [Function] PIPELINE DEBUG: Key-value pairs: {username=yddasdasda, status=Failed Login, ipAddress=192.168.2.140, entryPoint=Universal API}
2024-09-05T12:19:48.665+02:00 INFO [Function] PIPELINE DEBUG: EntryPoint: Universal API
2024-09-05T12:19:48.665+02:00 INFO [Function] PIPELINE DEBUG: IPAddress: 192.168.2.140
2024-09-05T12:19:48.665+02:00 INFO [Function] PIPELINE DEBUG: Status: Failed Login
2024-09-05T12:19:48.665+02:00 INFO [Function] PIPELINE DEBUG: Username: yddasdasda

When an actual log message is being processed:

2024-09-05T10:38:24.186+02:00 INFO [Function] PIPELINE DEBUG: username=dfsafasdasdasfrahda, status=Failed Login, ipAddress=192.168.2.140, entryPoint=Universal API
2024-09-05T10:38:24.187+02:00 INFO [Function] PIPELINE DEBUG: {}
2024-09-05T10:38:24.187+02:00 INFO [Function] PIPELINE DEBUG:
2024-09-05T10:38:24.187+02:00 INFO [Function] PIPELINE DEBUG:
2024-09-05T10:38:24.187+02:00 INFO [Function] PIPELINE DEBUG:

3. What steps have you already taken to try and solve the problem?
I tried using the debug log, giving me the desired values in the rule simulator, but not with the actual logs being processed, although the messages are the same.

Any help or tips would be appreciated. Also if any additional information is needed, please let me know.

Thanks & best regards

[Wine_Merchant]

Hey @bettels-uhi,

Should the kv_delimiters be ‘=’ and not ‘:’?

set_fields(

   fields:key_value(
   value: to_string($message.message),
   delimiters:",",
   kv_delimiters:"="
)
);