Pipeline - Simulator works, but the real thing does not

Graylog Central (peer support)
1

Hey guys,

yesterday I just started with pipelines and got a few to work, at least in the simulator.
I am using this message for the simulator:

simulatorMessage
simulatorMessage.png599×694 26.5 KB

this message is just a copy of a message I received in the Stream which is connected to my pipeline. (I checked multiple times if the pipeline is connected to the stream)
and this is my rule:

rule

In the simulator everything works fine and he creates a new field “myNr” with the value “51”. But the new messages in the Stream itself remain untouched.

Do I have to activate the Pipeline somewhere else or is it enough to add the connection in the “manage pipelines” menu?

Something strange I just saw is that the “Throughput” sometimes goes up to a few 100 msg/s, but I don´t know which ones he could have checked. The pipeline should only receive messages at full hours and even “all messages” does not received that much per second.

2

finally had time to test a bit more with the connections:
“myStream” - does not work
“all messages” - works
“myStream”, “all messages” - works

again only “myStream” and now it seems to work as well. I dont know why but i am glad that its all save and sound for now. Time to write rules :smiley:

3

system-config - Message Processors Configuration
what is your order?
You should do the stream first, and process pipeline next.

4
  1. AWS Instance Name Lookup
  2. GeoIP Resolver
  3. Pipeline Processor
  4. Message Filter Chain

this should be the standard :thinking:

5 
1. Message Filter Chain
2. Pipeline Processor
3. AWS Insstance Name Lookup (if not needed, disable it!)
4. GeoIP Resolver (if not needed, disable it!)

Please read https://blog.reconinfosec.com/geolocation-in-graylog/ (if you need geo ip)

2 Likes
6

Thanks for the hint :slight_smile:

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.