Pipeline Rules, order of execution (2022/v4.3 update)

[nisow95612]

This is an update to Rules, order of execution and 2nd half of Rename index set backing "All messages" stream? - #10 by tmacgbay

So far the answers are:

• All attached pipelines run in parallel. Normally all rules of stage X of attached pipelines execute at the same time (order of this processing is undefined), then all rules of stage X+1 execute and so on.

• remove_from_stream does not abort a pipeline, all attached pipelines will run to its finish as usual.

• add_to_stream marks the message for the new stream, but waits for all processing steps of currently attached pipelines to finish and then starts to run all the newly attached pipelines from their start.

• drop_message in graylog <= 4.2 just prevents message from being indexed into elasticsearch
• drop_message in graylog >= 4.3 additionally aborts its processing in running pipelines. Rules running in parallel in the same stage as drop_messages will run, but rules in all the following stages will be skipped.
• • Not sure about the extractors, though. If “message filter chain” is configured after pipeline processor I think it will still run?

• create_message and clone_message create independent messages (going through here and here), but this also waits for currently attached pipelines of the original message to return and then this new messages start their pipeline processing based on streams given to it by any add_to_stream(s).
• • Not sure about the extractors. If “message filter chain” is configured after pipeline processor, will extractors run for these new messages?

@mpfz0r

Also of Note: The stream rules are become a separate message processor:

This will make it more obvious that “stream rules” is just a one-time processing step that assigns streams. When a message stops matching a stream rule mid-pipeline, it does not matter.

[tmacgbay]

You can probably shift this to the misc section of the “Templates and Rules Exchange” as one of the cool tips located there.

Also one thing to note. If a message is sent to more than one pipeline at a time, the stages run in lockstep… more explicitly - if a message is pulled into two streams because both their stream rules match the message will always be processed by all pipelines in matching stage numbers. 1-1, then 2-2, then 3-3 etc. If one of the pipelines skips a stage the other pipelines have to finish and match up before they can all run together again (pipeline skipping has to wait).

It’s odd and I haven’t found a use case for it but that is how my experimentation turned out.

[nisow95612]

Done.
It’s mostly your research, after all. I just pulled it together and added the v4.3 part.
I should have used more words to describe that lockstep behavior - thanks, now it looks complete.

I think that behavior makes sense. I often put parsing rules at negative stages and sanitization and stream assignments in positive ones. That way I know that parsers won’t create new fields after sanitization… as long as I avoid add_to_stream, because that would match a new set of pipelines and start a new cycle of processing.