Key_value gummed up

[tmacgbay]

For some reason set_fields() is not working after my key_value() function and I can’t find that one thing causing an error. Maybe you can? Here is the scenario.

I am running a powershell command that that inserts a text line into a local file:

'winlog_event_id=4720,test_note=Testing-EventID-4720-NewUser,winlog_event_data_TargetUserName=Testing_user_name' | out-file $LogfilePath -append;pause

There is a filebeats configuration that picks it up that looks like this:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
   hosts: 
   - ${user.BeatsInput}
   ssl:
   verification_mode: none
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
 - test_eventID
processors:
  - add_fields:
      target: ''
      fields:
        winlog_api: wineventlog
        log_level: information
        winlog_channel: Security
        winlog_task: test_eventID
filebeat:
  inputs:
    - type: log
      paths:
        - C:\Program Files\Graylog\test_eventID.log

The message gets picked up properly via the beats input looking as it should:

winlog_event_id=4720,test_note=Testing-EventID-4720-NewUser,winlog_event_data_TargetUserName=Testing_user_name

It is then put on the pipeline and hits my “Testing eventID - set fields” key_value() rule first before any other rules:

rule "Testing eventID - set fields"
when
    contains(to_string($message.tags),"test_eventID") 
then
    let muffin       = to_string($message.message);
    let keyv_results = key_value(
        value:               muffin                      ,
        delimiters:          ","                         ,
        kv_delimiters:       "="                         ,
        ignore_empty_values: true                        ,
        allow_dup_keys:      false                       ,
        handle_dup_keys:     "take_first"                ,
        trim_key_chars:      "\""                        ,
        trim_value_chars:    "\""
    );
    debug(concat("++++ Results: ", to_string(keyv_results)));
    set_fields(keyv_results);
  
end

and when debug pops the keyv_results in the Graylog logs, it looks like a map:

2022-02-24T10:16:55.504-05:00 INFO  [Function] PIPELINE DEBUG: ++++ Results: {winlog_event_id=4720, test_note=Testing-EventID-4720-NewUser, winlog_event_data_TargetUserName=Testing_user_name}

But the set_fields() function that follows is NOT creating the appropriate fields! (There is some unneeded fluff in there I added for testing)

Grrr.

Can you spot the errant quote or some similar issue that is eluding me??

[tmacgbay]

De_gummed.

It was a simple log file, that was all… That was all is was supposed to be… you know… ending in .log and all.

I came across a post here that talked about powershell encocoding and noticed that when I set my command to encode in ascii that the results in the .log file somehow came out in Chinese. (WTF!) so I messed around and researched with encoding a bit more and then changed my .log file to a .txt file and now there is a better expectation of what the text going in and coming out will be… I had always scoffed at bigendian and littleendian… but after having been bitten for a day on “encoding”… I will find a new thing to scoff.

I changed the file type to .txt and made sure that powershell was encoded UTF8

"My log message" | Out-file $logFilePath -encoding UTF8 -append

so long … on such a small thing…