Can we modify a map object in a pipeline rule?

Graylog Tech Challenges
1

I am parsing a string with key_value():
let kv = key_value(to_string($message.message), " “, “=”, true, true,“take_last”,”",""");

I can access map values with kv.some_field, for example:
let another_field = do_sth(kv.some_field);

Now I’d like to do something like this:
kv.some_field = do_sth(kv.some_field);
set_fields(kv);

Is this somehow possible?

  • Daniel
2

Hello,

Not really understanding what you want besides parsing a string with key_value.
What is the outcome you are looking for in this pipeline?
Are you trying to manipulate the World Map widget? if so, I haven’t been able to do this yet.

3

No, this is not related to my world map post :slight_smile: Several things came into my mind, but maybe I didn’t get it right. Basically I have a string like this:

message: key1=“some string” key2=17, key3=“another string”
let kv = key_value(to_string($message.message), " “, “=”, true, true,“take_last”,”",""");

if I parse this, I’d get three new string values written to elastic, right?
key1: “some string”
key2: “17”
key3: “another string”

I could create a second stage that removes key2 and then adds it again after casting it to_long(). But elastic would treat this still as a string (because the first time it was written to elastic, it was a string), right? I would have to write it to a new field and then delete the old field.

Another thing is: do a lookup with the value from key3 and use this in the output:

  1. Parse the string into a map
  2. do the lookup
  3. modify the value
  4. call set_fields()
  5. done

This would come easier to my mind than:

  1. Parse the string into a map
  2. call set_fields()
  3. go to next stage
  4. grab the field in question again
  5. modify the value or take the value and write to a new field
  6. optional: remove the original field
  7. done

I imagine that modifying the field in the first place would be faster, too. Furthermore (maybe as an alternative), it would be cool if we could remove an entire key from the map, before it gets written. This would also handle my use cases.

In short: There is nothing I couldn’t do right now, but for me it would be easier and faster if I could mess with my map before it gets written.

4

Hello,
I just want to sum it up for a better understanding.
You have pipeline that extracts 3 different parts of the string and creates three different fields?

key1: “some string”
key2: “17”
key3: “another string”

Then create a second rule to remove key2 field and then add it back to the message field?

If this is correct it seems like a lot of work.

Have you tried to create an extractor using a Regular expression?

key3=(\s*(\S+)+\s*(\S+))

Using the cut method instead of copy.

image
image1283×629 39.7 KB

I can do simply pipeline rules but this may need someone else in the community to help out.
Also found this post.

Hope that helps

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.