JOIN GRAYLOG FOR OUR

ANNUAL CONFERENCE

Thursday,
October 21, 2021
10am-5pm CT
REGISTER NOW

Strange Pipelines behavior on Cluster

Strange Pipelines behavior on Cluster

On my single node Graylog, I have build for every input an own index,
so that the Windows logs has an own index, the Linux server has an own index and so on.
I managed this with streams.
Then I add pipelines rule to sort out some eventid’s to a “trash index” and added it to the index stream (index stream, sends inputs to index).
That works also.

So I build the same on the cluster.
But if I now add the pipeline rule to the index stream, nothing happens.
If I putt the pipeline rule to all messages, it works.

Why does it work with all messages and not with the index stream on the cluster?

All settings on the cluster are exact the same as the single node, I have checked this a few times with my teammates.

Stream01

rule "Move TO Index per Pipeline Rule"
when
contains(to_string($message.EventID), "5154") 
OR contains(to_string($message.EventID), "5156")
OR contains(to_string($message.EventID), "5158")
OR contains(to_string($message.EventID), "5447")
OR contains(to_string($message.EventID), "4656")
OR contains(to_string($message.EventID), "4658")
OR contains(to_string($message.EventID), "4660")
OR contains(to_string($message.EventID), "4663")
OR contains(to_string($message.EventID), "4674")
OR contains(to_string($message.EventID), "4690")
OR contains(to_string($message.EventID), "5140")
OR contains(to_string($message.EventID), "5145")
then
route_to_stream(name: "INDEX_TRASH", remove_from_default: true);
remove_from_stream(name: "INDEX_WINDOWS");
end

Operating system information

  • Ubuntu 20.04

Package versions

  • Graylog = Graylog 4.1.5+01c9198
  • MongoDB = v4.0.27
  • Elasticsearch = 7.10.2

Thanks for help.
Sven

What is the Message Processors Configuration on both? Do they match?

processbuffer_processors = 7
outputbuffer_processors = 7

are on both the same.

Sorry, meant this one below. Pipelines seem to have issues if you don’t have the Message Filter Chain before the Pipeline processor. Is it the same both areas?

1 Like

yes they are active but in diferent orders

  • singele Graylog
    Processor |Status|
    1 AWS Instance Name Lookup |active|
    2 Message Filter Chain |active|
    3 GeoIP Resolver |active|
    4 Pipeline Processor |active|
  • Cluster
    Processor |Status|
    1 AWS Instance Name Lookup |active|
    2 GeoIP Resolver |active|
    3 Pipeline Processor |active|
    4 Message Filter Chain |active|

After changing it to the order of the Singelnode it works.

Thx for your help tmacgbay

1 Like