Pipeline Processing - Parsing and Tweaking Field Names from Delimited String

[engageant]

1. Describe your incident:
I’m attempting to use a pipeline rule to extract additional fields from the http_uri_query field from MS Exchange ActiveSync IIS logs. The http_uri_query field contains a string like Error:NMStolen_SC1:1_PrxFrom:10.10.10.1_Ver1:161_HH:contoso.com_

There are multiple key/value pairs in the format Key:Value_, where _ is the delimiter between pairs.
2. Describe your environment:

• OS Information: Ubuntu 20.04

• Package Version: 4.3.9

3. What steps have you already taken to try and solve the problem?
It’s easy enough to use set_fields() with key_value() function to extract them, but I’m trying to take it one step further and convert the key (read: field name) portion to lowercase:

error:NMStolen_sc1:1_prxfrom:10.10.10.1_ver1:161_hh:contoso.com_

I’m aware that I can’t do this using the key_value() function, so I’m trying to use regex_replace() first:.

let renamed_fields = regex_replace("((?:|[^:_])*):", to_string($message.http_uri_query), lowercase("$1:"));

This pattern matches, but unfortunately the lowercase() function doesn’t work, and the net result is that renamed_fields is the same as $message.http_uri_query.

4. How can the community help?
Can this be accomplished without having to use rename_field()? This challenge isn’t just limited to setting field names (although I’m open to solving it in the context of field names); it appears to be a limitation of the regex_replace() function.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[gsmith]

Hey @engageant

I’m looking over this, but don’t have time right now. Just want you to know someone seen you post.

[gsmith]

Hey,
I cam across this ,

let renamed_fields = regex_replace("((?:|[^:_])*):", to_string($message.http_uri_query), lowercase("$1:"));

If you have the Field need alread you could use somethinglike this on an extractor with a converter.

As for the pipline I have this example.

then
  let regex = regex("(date=.*)",to_string($message.message));

  // Replace values with quotes to format "key":"value"
  let replace1 = regex_replace("([a-z0-9\\_\\-]+)=(?:\")([^\"]+)(?:\")", lowercase(regex["0"]), "$1");

[engageant]

The fields are generated by the pipeline, which for us occurs after the extractors in the chain so that wouldn’t work. For the pipeline example, unless I’m mistaken that’s only going to work for a single field.

Additional testing shows that running

to_string($message.http_uri_query), lowercase("FOO$1:"));

will lowercase “foo”, but not the regex group $1. Perhaps a bug, perhaps a feature?

[tmacgbay]

If you are just working on it in a rule you could use this:

let  get_low = lowercase(to_string($message.http_uri_query));

You can play around with get_low in the rule or change it permenantly for the message with:

set_field("http_uri_query", get_low);