Pipeline Rules, order of execution (2022/v4.3 update)

This is an update to Rules, order of execution and 2nd half of Rename index set backing "All messages" stream? - #10 by tmacgbay

So far the answers are:

  • drop_message in graylog <= 4.2 just prevents message from being indexed into elasticsearch
  • drop_message in graylog >= 4.3 additionally aborts its processing in running pipelines. Rules running in parallel in the same stage as drop_messages will run, but rules in all the following stages will be skipped.
    • Not sure about the extractors, though. If “message filter chain” is configured after pipeline processor I think it will still run?
  • create_message and clone_message create independent messages (going through here and here), but this also waits for currently attached pipelines of the original message to return and then this new messages start their pipeline processing based on streams given to it by any add_to_stream(s).
    • Not sure about the extractors. If “message filter chain” is configured after pipeline processor, will extractors run for these new messages?

This will make it more obvious that “stream rules” is just a one-time processing step that assigns streams. When a message stops matching a stream rule mid-pipeline, it does not matter.

3 Likes

You can probably shift this to the misc section of the “Templates and Rules Exchange” as one of the cool tips located there.

Also one thing to note. If a message is sent to more than one pipeline at a time, the stages run in lockstep… more explicitly - if a message is pulled into two streams because both their stream rules match the message will always be processed by all pipelines in matching stage numbers. 1-1, then 2-2, then 3-3 etc. If one of the pipelines skips a stage the other pipelines have to finish and match up before they can all run together again (pipeline skipping has to wait).

It’s odd and I haven’t found a use case for it but that is how my experimentation turned out.

2 Likes

Done.
It’s mostly your research, after all. I just pulled it together and added the v4.3 part.
I should have used more words to describe that lockstep behavior - thanks, now it looks complete.

I think that behavior makes sense. I often put parsing rules at negative stages and sanitization and stream assignments in positive ones. That way I know that parsers won’t create new fields after sanitization… as long as I avoid add_to_stream, because that would match a new set of pipelines and start a new cycle of processing.

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.