Where are pipelines in the processing stack


(Jake Smith) #1

Dear All,

My question is a simple one. If you look at the streams documentation at http://docs.graylog.org/en/2.3/pages/streams.html

There is a section showing how streams are processed internally. Given that a stream is directly added to an input as a stream_id field and the data is then send to the filter chain. Where are pipelines in this process, they are not shown in the filter part of the diagram?

Kind Regards

Jake


(Jochen) #2

Depending on the order of message processors in your Graylog cluster, the processing pipelines either run before or after the (legacy) Filter chain.

Check System / Configurations / Message Processors Configuration in the web interface.


(Jake Smith) #3

Hi Jochen,

My order is as follows:

1 - AWS INstance Name Lookup
2 - Message Filter Chain
3 - Pipeline Processor
4 - Geo-IP resolver

This would indicate that pipelines are processed after the Message Filter Chain and before the GEO-IP resolver. This would also mean things like extractors have also been run.

This would mean that the stream_id is already set before the pipeline processing begins which is the way that I want to use Graylog.

If I am correct please close thread

Jake


(Jochen) #4

Of course you can also add or remove a message from streams in your pipeline rules:


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.