Hi,
First let me thanks you for the great job you have done with graylog, it’s awesome.
I need some lights on rules execution in pipelines, as I can’t get what I expect (but I could be totally wrong on my expectations!).
According to the docs, Stages are used to control order and flow in pipelines.
But inside one stage, in what order are rules executed?
Could rules declaration order be reliable for execution order?
Also, inside the same stage, if a message is modified by an Action in a rule, does others rules Conditions could detect this change ?
Let me explain a bit:
I want to parse messages from a stream with many GROK patterns but to limit load I want to stop parsing when a GROK match.
Since alternation of nested GROK patterns is not fully supported in graylog (some fields are removed if they have the same name in nested patterns, see https://github.com/Graylog2/graylog2-server/issues/2029 ), I try to parse them serially and use a temporary field as flag to signal a match and skip following rules Actions.
But it seems that when a rule Action modify the message (by setting a flag field $message.grokked to true in my example), next rule’s Condition can’t see the change and act as it was false (then execute it’s Action and parse message again…).
Here is an example to illustrate my poorly written explanation:
I have a pipeline like this (simplified):
pipeline "GROK parsing messages"
stage 0 match either
rule "SET_GROKKED_FALSE";
stage 1 match either
rule "grok_pat1";
rule "grok_pat2";
stage 2 match either
rule "REMOVE_FIELD_IF_GROKKED";
Rules definitions:
rule "SET_GROKKED_FALSE"
when
true
then
set_field("grokked", false); // set flag field to false => not parsed yet
end
rule "grok_pat1"
when
to_bool($message.grokked) == false // if message didn't match a pattern yet..
then
set_fields(grok("%{S1_PATTERN1}", to_string($message.message), true)); // try to grok it
set_field("grokked", has_field("field_extracted_from_pattern1")); // if extracted field exist, flag is set to true
end
rule "grok_pat2"
when
to_bool($message.grokked) == false // if message was previously grokked, should be true so skip Action
then
set_fields(grok("%{S1_PATTERN2}", to_string($message.message), true));
set_field("grokked", has_field("field_extracted_from_pattern2"));
end
rule "REMOVE_FIELD_IF_GROKKED"
when
to_bool($message.grokked) == true // if message has been grokked,
then
remove_field("grokked"); // remove flag field.
end
So, if a message match GROK pattern in first rule:
-
Expected behavior
Message get new fields created with GROK, flag set to true.
Any other following rules Condition evaluate to false, thus never execute their Actions.
Flag field is removed in the end. -
Seen behavior
Message get new fields created with GROK, flag set to true.
BUT other rules evaluate it again as false, so they run their Actions. As their patterns didn’t match, new fields are not created but flag get overwrited to false.
Flag field is left with false value.
It looks like all rules Conditions in a stage are evaluated on Message in parallel, just when it enter stage and before any subsequent Action take place or not.
If I create a separate stage for each rules, it works as expected (but I doubt that’s the right way to do it).
I could be completely wrong with my code or maybe that’s the expected behavior, but I’m stuck on this for too long now and I’m lost, please help me a bit! ;o)
Thanks you.