Pipeline Moves messages, but still in All Messages Stream

[ThomasPowers (Tom Powers)]

Hello Everyone,

Been working with Pipelines and simple rules…which has been great.

But in this case, I am using a simple pipeline to move Windows event messages from the Sysmon log to my Sysmon stream. The goal is that we are breaking up our messaging into streams to get around the 1000 field index issues on our larger clients. The sysmon stream in this case is setup, with it’s own index applied to it, and messages are routing nicely. We have checked the box in the Sysmon stream to Remove Matches from the "All Messages’ stream.
The problem is that we have messages in the Sysmon stream as we would like, yet they are still in the All Messages stream. Obviously I am missing a step somewhere.

The pipeline has 2 simple rules

The first finds the Event Log for Sysmon

rule "sysmon logs"
when
** contains(to_string($message.Channel), “Microsoft-Windows-Sysmon/Operational”, true) **
then
end

then the next moves it to the Sysmon Stream, which has it’s own index setup

rule "Routes to Sysmon Stream"
when
** true**
then
** route_to_stream(name:“Sysmon”);**
end

This pipeline is executing from the All messages stream

All insight is appreciated

Thanks

TP

[ThomasPowers (Tom Powers)]

I should note that we are running this on Version 2.3.2, on Ubuntu 16.04
No plugins installed
Processing only GELF messages from Windows Machines, sent via NXLog, to a single Input

Message processors chain is
Pipeline Processor
Message Filter
GeoIP

[jan (Jan Doberstein)]

He Tom,

you need to make use of the processing function remove_from_stream or use the flag remove_from_default for the route_to_stream function.

The Flag in the stream setting is used only when the routing is done via stream rules.

I guess you need to upgrade to have the above mentioned pipeline functions available.

[ThomasPowers (Tom Powers)]

Ahhh…makes perfect sense! I upgraded to latest and put the remove_from_stream field in the rule and viola…works great!

Thanks

TP