Drop debug logs with pipeline

[xdsgkb (John)]

My 6 ESXi hypervisors floods graylog with 2M messages a day, majority of logs are DEBUG (~90%), so I want to drop DEBUG logs entirely.

SETUP
I have separate input just for ESXi: syslog on 1514/tcp.

I created “ESXi” stream which takes logs just from that input: Field gl2_source_input must match exactly 5XXXXXXXXXXX

Pipeline rule:
rule “dropOnDebug”
when
to_string($message.level) == “7”
then
drop_message();
end

Pipeline connections: stream “ESXi”

Stage0: dropOnDebug (At least one of the rules on this stage matches the message)

I still get DEBUG logs.

[jan (Jan Doberstein)]

• what is your processing order in System > configuration?
  • place processing pipeline after message filter chain
• does your messages have the field level and that contains a number?

[xdsgkb (John)]

Placing “Pipeline Processor” after “Message Filter Chain” in System > configuration solved the problem.
Does drop_message() function in pipeline deletes message from disk?

[jan (Jan Doberstein)]

Does drop_message() function in pipeline deletes message from disk?

To be exact this prevents the messages from being stored into Elasticsearch because they are dropped before they are ingested to it.