Can pipelines be used to redurce incomming traffic?

I have a question about pipelines; I read the documentation and searched the forum, but it seems I am still missing something. Most likely, this is something I should have been able to get from the documentation, but somehow, I just dont.

We have some devices that do send a lot of logs - much more than we want to log. Unfortunately, it is not possible to configure them to be less excessive. This is especially unfortunate as in some license modules the amount of incoming traffic as registered by graylog determines the actual price.

With pipelines I can block or remove messages. I can attach them to streams, and - as far as I understood - they are not written to the indices (and therefore not to elasticsearch). But is traffic filtered that way counted into the registered, incoming traffic of graylog? Or does it not count to it?

Hey @oebhardt

You can drop messages/logs using piepline, but even better if you can configure you log shippers to send only logs/messages you perfer.

1 Like

Hello @gsmith
thanks for the answer; yes, I am completely with You, it would be much better to avoid the traffic at all by configuring those devices to send only what I really want; unfortunately, this is not always possible.

So, just to clarify: if I drop traffic at a pipeline, it is not counted into the outgoing traffic used for licensing? While not ideal, it would be a great solution for some use cases!


I believe this would depend on where the pipeline is placed ( i.e., stage). so long as it doesnt hit Elasticsearch you should be good. TBH @tmacgbay probable would know better then I on these configrations and what to do.


rule “Drop Syslog Messages”
   has_field("level") AND  ( to_string($message.level) == “7” OR  lowercase(to_string($message.level)) == "debug" )

Or something like this.

Log shippers, there are setting/s to prevent specific logs from being shipped.
With Firewall’s/ Routers/Switchs/ AP’s, etc… most if not all have setting to minumize the amount of logs being sent.

A good example would be Cisco switch.

sw#configure terminal
sw#logging enable
sw#logging Graylog-Server
sw#logging trap level
	informational (default level)

@gsmith is right - only traffic going to ES is counted against the license.
From the docs:

Graylog has four counters; the last is counted for the licensed traffic.

  • org.graylog2.traffic.input
    The incoming message without any decoding; what is written to the journal before processing.
  • org.graylog2.traffic.decoded
    The message after the codec of the input has parsed the message (e.g. Syslog parser).
  • org.graylog2.traffic.system-output-traffic
    Traffic from archive restores; currently stored in memory only.
  • org.graylog2.traffic.output
    What is written to Elasticsearch after all processing is complete.

Graylog only measures Elasticsearch output. Measurement occurs when messages are serialized to Elasticsearch. If a message is written to multiple indices, the message will count for each index. It does not matter how many copies (replicas) the index has configured, as this is done in Elasticsearch.

Thanks @patrickmann,
today I’m at a customer site, I will implement it tomorrow.

i was thinking if i have syslog server we can that stripping the logs.
but graylog as log server as syslog server - then what is the best way here ?

Hey @ramindia

Not sure if I follow you, Correct me if I’m wrong but you have a Graylog server sending messages to another Syslog server?

take example cisco device directly sends logs to the Graylog server, there is no other Syslog server in the path.
how do we discard the messages that we don’t like ? or do not match the rules? ( because that is nonsense messages just occupied more space of SSD).

before I used to use syslog-ng to send to ELK (in logstash we can discard the data, before it go to elasticsearch) now I moved to Graylog platform.


You reffering to this?

There are a few shippers you can use, Standalone or use Graylog-Sidecar.
The different type are shown here

yes, i am referring that debug logs.

yes, I have seen sidecar yesterday I was reading the document,
but I am more focusing on Firewall and network devices, where we can not install any other tools
expect to use vendor-provide logging config.

But thank you I got the concept what you were mentioned…good for now.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.