Drop debug logs with pipeline


(John) #1

My 6 ESXi hypervisors floods graylog with 2M messages a day, majority of logs are DEBUG (~90%), so I want to drop DEBUG logs entirely.

SETUP
I have separate input just for ESXi: syslog on 1514/tcp.

I created “ESXi” stream which takes logs just from that input: Field gl2_source_input must match exactly 5XXXXXXXXXXX

Pipeline rule:
rule “dropOnDebug”
when
to_string($message.level) == “7”
then
drop_message();
end

Pipeline connections: stream “ESXi”

Stage0: dropOnDebug (At least one of the rules on this stage matches the message)

I still get DEBUG logs.


(Jan Doberstein) #2
  • what is your processing order in System > configuration?
    • place processing pipeline after message filter chain
  • does your messages have the field level and that contains a number?

(John) #3

Placing “Pipeline Processor” after “Message Filter Chain” in System > configuration solved the problem.
Does drop_message() function in pipeline deletes message from disk?


(Jan Doberstein) #4

Does drop_message() function in pipeline deletes message from disk?

To be exact this prevents the messages from being stored into Elasticsearch because they are dropped before they are ingested to it.


(system) closed #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.