Does Graylog 7 gracefully handle improper keys in flatten_json?

[WildHunter854 (Marc)]

1. Describe your incident:

Hello folks,

I need your help prior to opening a github issue, because our installation is fairly old (unmaintained 6.1) and we don’t have the possibility to check on a 7.0 Graylog instance if it behaves the same.

I would like to know if flatten_json() can now handle keys which contains invalid characters (a bit like Do slashes in json keys break flatten? but not the same use case)

More specifically, we would like to have each key name containing slashes, dots etc. to go through a fallback, deterministic transformation

[the idea is NOT for the key to be used verbatim into the opensearch store]

2. Describe your environment:

• OS Information: RHEL9

• Package Version: 6.1.something

• Service logs, configurations, and environment variables: n/a

3. What steps have you already taken to try and solve the problem?

• Ensured pipeline processing is working (correct order)
• Creating a pipeline rule, switched to Use Source Code Editor, then:
• Rule source:

rule "testing"
when
true
then
  let jnode = flatten_json(value: to_string($message.message), array_handler: "flatten", stringify: false);
  let parsed_map = to_map(jnode);
  set_fields(fields: parsed_map);
end

• Selected Simple message, paste this sample:

{"keyA":"valueA", "keyB":{"keyB1":"B1", "keyB2":"B2"}, "keyC":{"key/C1":"C1", "keyC2":"C2"}}

• Clicked [Run rule Simulation]
• Message is parsed as:

keyA
valueA

keyB_keyB1
B1

keyB_keyB2
B2

keyC_keyC2
C2

Notice how C1 is missing (because “key/C1” contains an invalid key for opensearch storage)

If you decide to save the rule and use it in a real pipeline. The server.log should output “[Message] Ignoring invalid or reserved key keyC_key/C1 for message <guid>” for the offending message.

What we’d like to happen (could be an extra parameter to flatten_json, or a change in default behavior):

keyA
valueA

keyB_keyB1
B1

keyB_keyB2
B2

keyC_key_C1 << would-be normalization of key/C1
C1

keyC_keyC2
C2

Real use case: flattening kubernetes audit messages including “annotations”:{“authorization.k8s.io/decision”:…

4. How can the community help?

Can you please test this sample in Graylog 7.0 and check if the flatten_json parsing logic has the same behavior?

Thanks