Can pipelines be used to redurce incomming traffic?

[oebhardt (Ortwin Ebhardt)]

Hello,
I have a question about pipelines; I read the documentation and searched the forum, but it seems I am still missing something. Most likely, this is something I should have been able to get from the documentation, but somehow, I just dont.

We have some devices that do send a lot of logs - much more than we want to log. Unfortunately, it is not possible to configure them to be less excessive. This is especially unfortunate as in some license modules the amount of incoming traffic as registered by graylog determines the actual price.

With pipelines I can block or remove messages. I can attach them to streams, and - as far as I understood - they are not written to the indices (and therefore not to elasticsearch). But is traffic filtered that way counted into the registered, incoming traffic of graylog? Or does it not count to it?

[gsmith]

Hey @oebhardt

You can drop messages/logs using piepline, but even better if you can configure you log shippers to send only logs/messages you perfer.

[oebhardt (Ortwin Ebhardt)]

Hello @gsmith
thanks for the answer; yes, I am completely with You, it would be much better to avoid the traffic at all by configuring those devices to send only what I really want; unfortunately, this is not always possible.

So, just to clarify: if I drop traffic at a pipeline, it is not counted into the outgoing traffic used for licensing? While not ideal, it would be a great solution for some use cases!

[gsmith]

Hey,

@oebhardt

So, just to clarify: if I drop traffic at a pipeline, it is not counted into the outgoing traffic used for licensing?

I believe this would depend on where the pipeline is placed ( i.e., stage). so long as it doesnt hit Elasticsearch you should be good. TBH @tmacgbay probable would know better then I on these configrations and what to do.

Example:

rule “Drop Syslog Messages”
when
   has_field("level") AND  ( to_string($message.level) == “7” OR  lowercase(to_string($message.level)) == "debug" )
then
   drop_message();
end

Or something like this.

Pipeline Rule - Drop messages with field containing IP

you can use the debug() function and watch what it’s output is in the Graylog logs: tail -f /var/log/graylog-server/server.log rule "Drop NAT from specific Subnet" when starts_with(to_string($message.Int_Src_IP), "172.16.40") then debug("Source IP match:"); debug(to_string($message.Int_Src_IP)); drop_message(); end if nothing shows you can change the when to has_field("Int_Src_IP") just to make sure you get to the debug Debug Function - Graylog 4.0

Log shippers, there are setting/s to prevent specific logs from being shipped.
With Firewall’s/ Routers/Switchs/ AP’s, etc… most if not all have setting to minumize the amount of logs being sent.

A good example would be Cisco switch.

sw#enable
sw#configure terminal
sw#logging enable
sw#logging Graylog-Server
sw#logging trap level
	emergencies
	alerts
	critical
	errors
	warnings
	notifications
	informational (default level)
	debugging
#end

[patrickmann]

@gsmith is right - only traffic going to ES is counted against the license.
From the docs:

Graylog has four counters; the last is counted for the licensed traffic.

• org.graylog2.traffic.input
  The incoming message without any decoding; what is written to the journal before processing.
• org.graylog2.traffic.decoded
  The message after the codec of the input has parsed the message (e.g. Syslog parser).
• org.graylog2.traffic.system-output-traffic
  Traffic from archive restores; currently stored in memory only.
• org.graylog2.traffic.output
  What is written to Elasticsearch after all processing is complete.

Graylog only measures Elasticsearch output. Measurement occurs when messages are serialized to Elasticsearch. If a message is written to multiple indices, the message will count for each index. It does not matter how many copies (replicas) the index has configured, as this is done in Elasticsearch.

[oebhardt (Ortwin Ebhardt)]

Thanks @patrickmann,
today I’m at a customer site, I will implement it tomorrow.

[ramindia (R!)]

i was thinking if i have syslog server we can that stripping the logs.
but graylog as log server as syslog server - then what is the best way here ?

[gsmith]

Hey @ramindia

@ramindia

but graylog as log server as syslog server - then what is the best way here ?

Not sure if I follow you, Correct me if I’m wrong but you have a Graylog server sending messages to another Syslog server?

[ramindia (R!)]

take example cisco device directly sends logs to the Graylog server, there is no other Syslog server in the path.
how do we discard the messages that we don’t like ? or do not match the rules? ( because that is nonsense messages just occupied more space of SSD).

before I used to use syslog-ng to send to ELK (in logstash we can discard the data, before it go to elasticsearch) now I moved to Graylog platform.

[gsmith]

Hey

@ramindia

how do we discard the messages that we don’t like ? or do not match the rules?

You reffering to this?

@gsmith

rule “Drop Syslog Messages”
when
   has_field("level") AND  ( to_string($message.level) == “7” OR  lowercase(to_string($message.level)) == "debug" )
then
   drop_message();
end

@ramindia

before I used to use syslog-ng to send to ELK

There are a few shippers you can use, Standalone or use Graylog-Sidecar.
The different type are shown here

[ramindia (R!)]

yes, i am referring that debug logs.

yes, I have seen sidecar yesterday I was reading the document,
but I am more focusing on Firewall and network devices, where we can not install any other tools
expect to use vendor-provide logging config.

But thank you I got the concept what you were mentioned…good for now.