Rules execution order in stages and changes visibility?


(Zenix) #1

Hi,

First let me thanks you for the great job you have done with graylog, it’s awesome.

I need some lights on rules execution in pipelines, as I can’t get what I expect (but I could be totally wrong on my expectations!).

According to the docs, Stages are used to control order and flow in pipelines.
But inside one stage, in what order are rules executed?
Could rules declaration order be reliable for execution order?

Also, inside the same stage, if a message is modified by an Action in a rule, does others rules Conditions could detect this change ?
Let me explain a bit:

I want to parse messages from a stream with many GROK patterns but to limit load I want to stop parsing when a GROK match.
Since alternation of nested GROK patterns is not fully supported in graylog (some fields are removed if they have the same name in nested patterns, see https://github.com/Graylog2/graylog2-server/issues/2029 ), I try to parse them serially and use a temporary field as flag to signal a match and skip following rules Actions.

But it seems that when a rule Action modify the message (by setting a flag field $message.grokked to true in my example), next rule’s Condition can’t see the change and act as it was false (then execute it’s Action and parse message again…).

Here is an example to illustrate my poorly written explanation:

I have a pipeline like this (simplified):

pipeline "GROK parsing messages"
stage 0 match either
  rule "SET_GROKKED_FALSE";
stage 1 match either
  rule "grok_pat1";
  rule "grok_pat2";
stage 2 match either
  rule "REMOVE_FIELD_IF_GROKKED";

Rules definitions:

rule "SET_GROKKED_FALSE"
when
    true
then
	set_field("grokked", false);			// set flag field to false => not parsed yet
end

rule "grok_pat1"
when
    to_bool($message.grokked) == false 		// if message didn't match a pattern yet..
then
    set_fields(grok("%{S1_PATTERN1}", to_string($message.message), true));	// try to grok it
    set_field("grokked", has_field("field_extracted_from_pattern1"));		// if extracted field exist, flag is set to true
end

rule "grok_pat2"
when
    to_bool($message.grokked) == false 		// if message was previously grokked, should be true so skip Action
then
    set_fields(grok("%{S1_PATTERN2}", to_string($message.message), true));
    set_field("grokked", has_field("field_extracted_from_pattern2"));
end

rule "REMOVE_FIELD_IF_GROKKED"
when
    to_bool($message.grokked) == true		// if message has been grokked,
then
    remove_field("grokked");				// remove flag field. 
end

So, if a message match GROK pattern in first rule:

  • Expected behavior
    Message get new fields created with GROK, flag set to true.
    Any other following rules Condition evaluate to false, thus never execute their Actions.
    Flag field is removed in the end.

  • Seen behavior
    Message get new fields created with GROK, flag set to true.
    BUT other rules evaluate it again as false, so they run their Actions. As their patterns didn’t match, new fields are not created but flag get overwrited to false.
    Flag field is left with false value.

It looks like all rules Conditions in a stage are evaluated on Message in parallel, just when it enter stage and before any subsequent Action take place or not.
If I create a separate stage for each rules, it works as expected (but I doubt that’s the right way to do it).

I could be completely wrong with my code or maybe that’s the expected behavior, but I’m stuck on this for too long now and I’m lost, please help me a bit! ;o)

Thanks you.


(Jan Doberstein) #2

If I create a separate stage for each rules, it works as expected (but I doubt that’s the right way to do it).

currently that is the only option you have in Graylog.


(Zenix) #3

Ouch…
Thanks for your answer but that’s a bit disappointing that rules can’t interact with each others on a message. So for example if like me you have a long list of GROK patterns to parse various messages you have to apply them all, even if the first match…


(Jan Doberstein) #4

sorry that this feature is currently not present. But you can always add this as a feature request to github:

That will raise the possibility that this will be available in a future version.