Most efficient way to use pipelines for routing

Graylog Central (peer support)
,
1

Hello all,

I’m wondering what the most efficient way to route messages to streams is; i’ve been using a pipeline, attached to one input stream, this pipeline has 6 different rules.
Each rule evaluates a certain field, then, if the condition matches, it routes that message to a certain stream.

Would it be more effective to only have one routing rule per pipeline? That way all other rules wouldn’t be evaluated for each messages (each message comes into a ‘Global’ stream, then is routed to a substream, depending on what operation it is, messages are not required to be routed into multiple streams with these pipelines and rules).

Since there are only aggregated pipeline processing time metrics available (not processing time per pipeline), i’m not suire there’s a clear answer to this.

The input stream in question gets ~100 messages per second during normal hours, this will be increasing drastically so want to make sure i’m using the best approach to deal with these messages.

Thanks!

2

Hey @josh.7

Actually that sound like a Textbook Pipeline you described :joy: As with anything the more it is used, you may have to keep an eye out on resources ( CPU, Memory, etc…). There are Stages also that can be utilized.

Just an idea if you have different devices , say Firewalls, Windows, Linux , switches, etc… it wouldn’t be a bad Idea to separate them into there own Input. You can make index sets for each which would attach to different streams. The reason I say this is we started with One stream , default index set. Went from 5 -10 GB day. As we expanded the amount of logs grew to 100- 200GB a day. It became a pain to start reconfiguring our GL Cluster after that.