Pipeline rule, multiple values using regex function return an empty table

piellick · January 14, 2019, 8:34pm

Hi everyone,
here is a pipeline rule who normaly catch multiple quoted values like ‘xxxxx’

rule “quote extractor”
when
true
then
let result = regex(“‘.*?’”,to_string($message.message));
set_field(“quotedvalue_1”,to_string(result));
end

Here is a log message exemple:

hostname hlspullpusher.hlspusher/41497 zf103: ‘QUOTED VALUE 1’ - unable to push HLS fragment ‘QUOTED VALUE 2’ : Server error ‘QUOTED VALUE 3’|#warning,hlspusher,local4,support

!!! warning blockquote formatted text change the quote ’ to ‘ …warning if you copy paste the log message !!!

Tested with freeformater (Free Online Java Regular Expression Tester - FreeFormatter.com), regex is working well.

On pipeline simulator, i have en empty table :

i convert table to a string to have all values listed on the field, i try with

set_field(“quotedvalue_1”,result[“0”]);

i have an empty value too.

My graylog processors configuration :

I have thtoughput over my pipeline and rule :

Any ideas ?

macko003 · January 15, 2019, 1:41pm

The linked page doesn’t give me a match.
So you have to work on your regex pattern.
Have you tried the escape the ’ character?

piellick · January 16, 2019, 8:59am

Hi @macko003, the blockquote formatting change quote

the regexex:

‘.*?’

log entrie for test:

hostname hlspullpusher.hlspusher/41497 zf103: ‘QUOTED VALUE 1’ - unable to push HLS fragment ‘QUOTED VALUE 2’ : Server error ‘QUOTED VALUE 3’|#warning,hlspusher,local4,support

macko003 · January 16, 2019, 9:02am

Yes, but when I tested, it worked with escaped ’

piellick · January 16, 2019, 10:16am

Ok, il will try my rules by escaping quotes…good idea

piellick · January 16, 2019, 12:38pm

after somes test using debug() function on the rules … it look strange … the regex function catch nothing . i try with another “easy” regex.

“warning” value is available on the log line:

Rules on my pipeline:

rule “quote extractor”
when
true
then
let result = regex(“war[acn]ing”,to_string($message.message));
debug (result);
set_field(“quotedvalue_1”,to_string(result));
end

debug output :

2019-01-16 12:36:20,027 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-16 12:36:20,027 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-16 12:36:20,028 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-16 12:36:20,028 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-16 12:36:20,028 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-16 12:36:21,056 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}

i forgot something on my rules ?

I’m on

graylog 2.5.0

Using

UDP syslog input —> a stream —> 1 pipeline connected to and 1 rule

And my pipeline works well :

macko003 · January 17, 2019, 3:48pm

It seems to be ok.
Try debug more, to localize the problem.

eg.
use debug for the $message.message.
try regexp in the when part. ( regex(".*", to_string($message.name)).matches == true)(I use somethink like that in my pipeline)
etc

maybe it is a bug.

piellick · January 17, 2019, 4:22pm

Hi @macko003 thanks for your help …

It has the taste of a bug

last test (i tested with the “matches” on when condition too but without result) :

rule "Generic REGEX quote extractor"
when
    true
then
debug ($message.syslog_message);
let result = regex("\'.*?\'",to_string($message.syslog_message));
debug (result);
set_fields(result);
end

Debug() output:

2019-01-17 16:18:21,607 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: start archive cleaner on channel '2025279_multi'|#phanes,ott,support,local4
2019-01-17 16:18:21,607 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-17 16:18:21,611 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: start purge channel '2025279_multi' from '1' (1970-01-01 00:00:01) to '1547741586' (2019-01-17 16:13:06)|#phanes,ott,support,local4
2019-01-17 16:18:21,611 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}

Regex() is still catching nothing …

macko003 · January 17, 2019, 8:49pm

try regex “.*” only.
As far as I know in graylog you need to use double escape (\\') (I think I forgot it before, sorry)

piellick · January 22, 2019, 11:54am

HI @macko003, don’t worry i appreciat your help

i try with double escape, still nothing :

rule "Generic REGEX quote extractor"
when
    true
then
debug ($message.syslog_message);
let result = regex("\\'.*?\\'",to_string($message.syslog_message));
debug (result);
set_fields(result);
end

i try with “.*” in my regex, still nothing … strange

rule "Generic REGEX quote extractor"
when
    true
then
debug ($message.syslog_message);
let result = regex(".*",to_string($message.syslog_message));
debug (result);
set_fields(result);
end

Debug output:

2019-01-22 11:49:56,494 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: '2025279_multi-239.255.80.1:10004:lan6' - no data received (timeout is '3')|#error,mezzanine,local4,support
2019-01-22 11:49:56,494 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2019-01-22 11:49:56,494 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: '2025279_multi-239.255.80.1:10007:lan6' - no data received (timeout is '3')|#error,mezzanine,local4,support
2019-01-22 11:49:56,494 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}

macko003 · January 22, 2019, 12:02pm

i have mo more ideas

piellick · January 22, 2019, 12:20pm

me too … i will create a ticket on github. It’s strange.

benvanstaveren · January 22, 2019, 1:55pm

You may want to add grouping to your regex

And maybe change the regex to this: regex("('.*?')+", to_string($message.message)) - this will grab all quoted values at once, and you can then use result[0], result[1], etc. to grab them. They do include the ’ quote in the value, so you could then strip that off.

piellick · January 22, 2019, 2:50pm

that crazy thanks @benvanstaveren … regex catch now the first group but not the second.

rule "Generic REGEX quote extractor"
when
true
then
debug ($message.syslog_message);
let result = regex("('.*?')+",to_string($message.syslog_message));
debug (result);
set_fields(result);
end

2019-01-22 14:47:29,483 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: '2025279_multi-239.255.80.1:10004:lan6' - no data received (timeout is '3')|#error,mezzanine,local4,support

2019-01-22 14:47:29,483 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {0='2025279_multi-239.255.80.1:10004:lan6'}

2019-01-22 14:47:29,483 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: '2025279_multi-239.255.80.1:10002:lan6' - no data received (timeout is '3')|#error,mezzanine,local4,support

2019-01-22 14:47:29,483 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {0='2025279_multi-239.255.80.1:10002:lan6'}

2019-01-22 14:47:29,483 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: '2025279_multi-239.255.80.1:10003:lan6' - no data received (timeout is '3')|#error,mezzanine,local4,support

2019-01-22 14:47:29,484 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {0='2025279_multi-239.255.80.1:10003:lan6'}

crazy … regex function catch only the first group

benvanstaveren · January 22, 2019, 7:04pm

Heuh… okay, umm that’s weird because I could swear it should grab all - unless it requires the /g modifier to do it on the entire message but I’m sure java’s regex already does that…

piellick · January 23, 2019, 9:35am

Weird …for sure … i have no more idea

benvanstaveren · January 24, 2019, 8:36am

Yeah, I’m kind of out of ideas here too

piellick · January 24, 2019, 12:35pm

I could be helpful to have a push from the graylog team

piellick · January 28, 2019, 1:18pm

no one ?

Any help will be appreciated.

jan · January 28, 2019, 1:59pm

When you use regex and groups, you need to include what group you want to use.

rule "extract {content}"
when
    true
then
    let message=to_string($message.message);
   
    let ex = regex(pattern: "\\{(.*?)\\}", value: message);
    
    set_field("first", ex["0"]);
    set_field"("second", ex["1"]);
end

In this long read I wasn’t able to find a log entry that I could copy&paste or a pipeline rule I can copy&paste - so sorry no direct help.

Topic		Replies	Views
Pipeline rule, multiple values using regex function, only possible to return value ["0"] Graylog Central (peer support) pipeline-rules , debuggingpl	2	2218	April 27, 2020
Need some help with regex in pipeline rule Graylog Central (peer support) pipeline-rules	2	2260	July 7, 2023
Graylog pipeline regex multi match Graylog Central (peer support) pipeline-rules	5	4693	July 15, 2020
Pipelines and regex (and some extractor talk) Graylog Central (peer support) regex-special-charac	6	775	July 11, 2023
Issue with regex-array in pipeline-rule Graylog Central (peer support) pipeline-rules	3	1473	May 8, 2020

Pipeline rule, multiple values using regex function return an empty table

Related topics