Multi Part Rule within Streams

I have a stream that has a long list of rules within it each matching a winlogbeat_event_id.

MS Events are unique numbers within the log type they sit within, for example 1102 is Audit Log Cleared within Security Log and DFS Replication Stopped within DSFR Log.

How can I write a Stream rule that is equivalent to:

winlogbeat_event_id:4618 OR (winglogbeat_event_id:1102 AND winlogbeat_source_name:Security).

This is easy in Search and would be easy in Streams if I didn’t want to also have other events filtered into the same stream.

You can use pipeline rule like this to route to specific stream:

when
    has_field("winlogbeat_event_id") AND
    (to_long($message.winlogbeat_event_id) == 4618 OR (to_long($message.winlogbeat_event_id) == 1102 AND to_string($message.winlogbeat_source_name == "Security")))
then
    // Route the current processed message to a stream with ID `512bad1a535b43bd6f3f5e86` (preferred method)
    route_to_stream(id: "512bad1a535b43bd6f3f5e86");

    // Route the current processed message to a stream named `Custom Stream`
    route_to_stream(name: "Custom Stream");
end

https://docs.graylog.org/en/3.3/pages/pipelines/functions.html#route-to-stream

but a pipeline is connected to a specific stream, rather than processing messages before they hit a stream isn’t it?

So if I’ve already routed the messages to a stream (in this case Windows Security Logs) this would just take the messages within Windows Security Logs and redirect them to AD Events where really they need to be in both.

Should I be running the pipeline on all_messages and then using it to route to both the streams I’m interested in and remove the rules on the streams themselves?

An added issue is that without a Raw message I can’t test the Pipelines and Graylog doesn’t give you the means to extract a Raw message as its lost during processing.

you can go any route @nick

most people do stream routing on the all messages stream (removing the all message stream) what allows them to check the all message stream if they have not parsed messages.

In the targeted streams they send the messages to are stream rules in a later numbering to make some normalisation, enrichment or similar with the messages and maybe later routing to a third stream where the messages are finally stored or similar.

see the processing pipelines as rail tracks and the messages coming in and you route them between the different tracks with a switch but the messages can only move forward.

does that make sense to you?

Jan

If I were to use a single pipeline to process the all messages stream then that essentially replaces the Stream rules. Why have both?

It doesn’t look like you can pass a message from one pipeline to another, just between stages in a single pipeline? In your analogy what acts as the switch?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.