Strange Pipelines behavior on Cluster

[SR_CSV]

Strange Pipelines behavior on Cluster

On my single node Graylog, I have build for every input an own index,
so that the Windows logs has an own index, the Linux server has an own index and so on.
I managed this with streams.
Then I add pipelines rule to sort out some eventid’s to a “trash index” and added it to the index stream (index stream, sends inputs to index).
That works also.

So I build the same on the cluster.
But if I now add the pipeline rule to the index stream, nothing happens.
If I putt the pipeline rule to all messages, it works.

Why does it work with all messages and not with the index stream on the cluster?

All settings on the cluster are exact the same as the single node, I have checked this a few times with my teammates.

rule "Move TO Index per Pipeline Rule"
when
contains(to_string($message.EventID), "5154") 
OR contains(to_string($message.EventID), "5156")
OR contains(to_string($message.EventID), "5158")
OR contains(to_string($message.EventID), "5447")
OR contains(to_string($message.EventID), "4656")
OR contains(to_string($message.EventID), "4658")
OR contains(to_string($message.EventID), "4660")
OR contains(to_string($message.EventID), "4663")
OR contains(to_string($message.EventID), "4674")
OR contains(to_string($message.EventID), "4690")
OR contains(to_string($message.EventID), "5140")
OR contains(to_string($message.EventID), "5145")
then
route_to_stream(name: "INDEX_TRASH", remove_from_default: true);
remove_from_stream(name: "INDEX_WINDOWS");
end

Operating system information

• Ubuntu 20.04

Package versions

• Graylog = Graylog 4.1.5+01c9198
• MongoDB = v4.0.27
• Elasticsearch = 7.10.2

Thanks for help.
Sven

[tmacgbay]

What is the Message Processors Configuration on both? Do they match?

[SR_CSV]

processbuffer_processors = 7
outputbuffer_processors = 7

are on both the same.

[tmacgbay]

Sorry, meant this one below. Pipelines seem to have issues if you don’t have the Message Filter Chain before the Pipeline processor. Is it the same both areas?

[SR_CSV]

yes they are active but in diferent orders

• singele Graylog
  Processor |Status|
  1 AWS Instance Name Lookup |active|
  2 Message Filter Chain |active|
  3 GeoIP Resolver |active|
  4 Pipeline Processor |active|
• Cluster
  Processor |Status|
  1 AWS Instance Name Lookup |active|
  2 GeoIP Resolver |active|
  3 Pipeline Processor |active|
  4 Message Filter Chain |active|

After changing it to the order of the Singelnode it works.

Thx for your help tmacgbay