Syntax: regex replacement in Pipeline Rule

[Aksel]

Hi Folks,

i am trying to reformate a field that i already extracted with a grok pattern.
By this i am willing to replace an extractor (replace with regex) that i am using.

rule "R-DeviceID Regex Replacement"

when
    has_field("R-DeviceID")
then
    let N-DeviceID = regex_replace(pattern: "(\d)(/\d)\/(\d+)\/(\d+)\:(\d+)\.(\d+)\.(\d+)",
    value: to_string($message.R-DeviceID),
    replace:"$3-$4-$5-$6-$7");
    set_field = ("R-DeviceID", N-DeviceID);
end

Is something wrong with the Syntax i am using ?
The Value should be reformated from 1/1/01/02:2.1.1 to 01-02-2-1-1

Thanks in Advance Folks

[gsmith]

Hello && Welcome

I might be able to help. Just and FYI I’m not very good at pipelines /w regex but I did find some flaws.
It should look some thing like this.

rule "R-DeviceID Regex Replacement"

when
    has_field("R-DeviceID")
then
    let N-DeviceID = regex_replace("(\d)(/\d)\/(\d+)\/(\d+)\:(\d+)\.(\d+)\.(\d+),to_string($message.R-DeviceID), "$3-$4-$5-$6-$7");
    set_field = ("R-DeviceID", N-DeviceID);
end

Next, I copy & paste you configuration above in Pipeline rules and there is an error with the regex section, Invalid expression.

I was testing regex configuration that could work here https://regex101.com/

Regex you have used above.

One I have used.

But when replacing regex with the one I tested an error Invalid expression still shows .

Using this rule.

rule "R-DeviceID Regex Replacement"

when
    has_field("R-DeviceID")
then
    let N-DeviceID = regex_replace("[^<>,\s]+(\d+)\:(\d+)\.(\d+)\.(\d+),to_string($message.R-DeviceID), "$3-$4-$5-$6-$7");
    set_field = ("R-DeviceID", N-DeviceID);
end

Think were forgetting something but not sure what. @tmacgbay probably be our best bet to find out what this could be.

I found this post, it might help with the pipeline.

Replacing UID with Username using lookup table

I messed up - take out the first “user not found” that deletes the number you are looking for and screws the whole thing up! - I had deleted it in my test but messed up on the one I pasted in!!! so it should be: rule "Graylog Web Access User Convert" when contains(to_string($message.message),"UID=") then // create a local to make it slightly less intensive to work with let robin = to_string($message.message); let batman = replace(robin, "1914600003", "steveno"); let…

I just ran into something I totally forgot. There is a regex extractor called Replace with regular expression

[Aksel]

Hey Folks,

thanks for the quick reply.

As i mentioned in my first reply i already have an Extractor Replace with regular expression what i want to replace with a pipeline, because extractors use more system ressources than grok_pattern in a pipeline rule.

@gsmith

I just ran into something I totally forgot. There is a regex extractor called Replace with regular expression

The Regex i am using is just fine

I tried so many Syntaxes for the regex_replace in a pipeline all i get is: Invalid Expression.

Yes i know about these and i am using so many Extrators but i am about to replace all of them.

@gsmith

I was testing regex configuration that could work here https://regex101.com/

Regex you have used above.

I also tested the Regex on https://regex101.com/ by using Java Flavor because the regular expression to which the “value” string is to be matched; uses Java regex syntax.

[Aksel]

I was playing arround again and testing some alternatives.

To avoid the “Invalid expression” i have created a Grok pattern in System/Grok Patterns as a single word in the processing in the Piperule

Official Docs: regex-replace

rule "DHCPv6 R-DeviceID Regex Replacement"

when 
    has_field("R_DeviceID")
then
    let N_DeviceID = regex_replace("%{deviceid}",to_string($message.R_DeviceID),"$3-$4-$5-$6-$7");
    set_field("R_DeviceID", N_DeviceID);
end

i tried also

rule "DHCPv6 R_DeviceID Regex Replacement"

when 
    has_field("R_DeviceID")
then
    let N_DeviceID = regex_replace(pattern: "%{deviceid}",value: to_string($message.R_DeviceID),replacement: "$3-$4-$5-$6-$7");
    set_field("R_DeviceID", N_DeviceID);
end

I am getting this error message “Unable to pre-compute” value for 1st argument pattern in call to funtion regex_replace: Illegal repetition near index 0

[tmacgbay]

A couple of rule things:

• regex_replace doesn’t like the dash in field names so use quotes:

to_string($message."R-DeviceID")

• Missed and ending quote on the pattern:

regex_replace("[^<>,\s]+(\d+)\:(\d+)\.(\d+)\.(\d+)<RIGHT HERE!>,to_st...

• set_field() is a function and doesn’t need an = sign so it would be:

set_field("R-DeviceID", the_result);

• Regex in the pipeline needs double escapes (catches everyone)

Resulting non-complaining and slightly modified yet completely untested rule:

rule "R-DeviceID Regex Replacement"

when
    has_field("R-DeviceID")
then
    let the_result = regex_replace("(\\d+)\\/(\\d+)\\/(\\d+)\\/(\\d+)\\:(\\d+)\\.(\\d+)\\.(\\d+)",to_string($message."R-DeviceID"),"$3-$4-$5-$6-$7",true);
    set_field("R-DeviceID", the_result);
end

[gsmith]

Thanks for jumping in

@tmacgbay

pipeline needs double escapes (catches everyone)

@tmacgbay I just noticed you also used two \\ instead of one.

[Aksel]

Hey Folks && Thanks

rule "DHCPv6 R-DeviceID Regex Replacement"

when 
    has_field("R_DeviceID")
then
    let DeviceID = regex_replace("(\\d+)\\/(\\d+)\\/(\\d+)\\/(\\d+)\\:(\\d+)\\.(\\d+)\\.(\\d+)", to_string($message."R_DeviceID"),"$3-$4-$5-$6-$7",true);
    set_field ("R_DeviceID", DeviceID);
end

A little something still missing, maybe should i just change the Fieldname ?

[tmacgbay]

Use the the debug() function to figure out what is going on in your rule - you can watch for the results in your Graylog log:

tail -f /var/log/graylog-server/server.log

rule "DHCPv6 R-DeviceID Regex Replacement"

when 
    has_field("R_DeviceID")
then
    let DeviceID = regex_replace("(\\d+)\\/(\\d+)\\/(\\d+)\\/(\\d+)\\:(\\d+)\\.(\\d+)\\.(\\d+)", to_string($message."R_DeviceID"),"$3-$4-$5-$6-$7",true);
    debug(concat("****$message.R_DeviceID equals this: ",to_string($message."R_DeviceID")));
    debug(concat("****DeviceID results in this: ",DeviceID ));
    set_field ("R_DeviceID", DeviceID);
end

EDIT: Corrected to_string() in second param of first debug()

[Aksel]

Hello again,

i am getting an Error by this Line

debug(concat("****$message.R_DeviceID equals this: ",$message."R_DeviceID"));

Expected type String for argument second but found Object in call to function in line 7 pos10

[patrickmann]

I think it should be to_string($message."R_DeviceID") as the second param to concat

[Aksel]

yes it is
debug(concat("****$message.R_DeviceID equals this: ",to_string ($message.“R_DeviceID”)));

[Aksel]

first of all i want to thanks everybody for the Help.

The Rule is placed in the Pipeline in debuging mode, there is no output to see in the Logs, there is something wrong still with the choosen Regex .

The Field “R_DeviceID” and other Fields have been already extrated thru another Pipeline Rule using a Crok pattern.

[tmacgbay]

@Aksel

The Field “R_DeviceID” and other Fields have been already extrated thru another Pipeline Rule

…Hopefully in a previous stage… rules within a single stage are essentially run in parallel and can’t rely on each other for data…

If you are not seeing the debug() results, then there is no R_DeviceID field for the rule to see…

[Aksel]

Thanks for the Guiding and for the Help everybody!

Yes it is, the Regex Replacement should be placed in Stage 1 and the other Grok Pattern Rules should be in Stage 0.

(Stage priority. The lower the number, the earlier it will execute.)

[Aksel]

Recapitualtion and for Documentation for Others!

The Goal
Reformating this Value 1/1/01/02:2.1.1 to 01-02-2-1-1 after it has been parsed from message string by using Grok Pattern to many Fields one of them is named “R-DeviceID” and the Value 1/1/01/02:2.1.1 is stored in it.

Final Result

rule "DHCPv6 R-DeviceID Regex Replacement"

when 
    has_field("R_DeviceID")
then
    let DeviceID = regex_replace("(\\d+)\\/(\\d+)\\/(\\d+)\\/(\\d+)\\:(\\d+)\\.(\\d+)\\.(\\d+)", to_string($message."R_DeviceID"),"$3-$4-$5-$6-$7",true);
    set_field ("R_DeviceID", DeviceID);
end

Regex Replacement should be placed in Stage 1 and the other Grok Pattern Rules should be in Stage 0.
(Stage priority. The lower the number, the earlier it will execute.)

Thanks to @tmacgbay @gsmith @patrickmann for Helping to find a Solution.