Dumping unwanted data from the start to reduce load

#1

I would like to reduce storage and workload by dumping unwanted syslogs as early as possible. Ideally before they are ever written to storage and before any extractors or additional rules are run against them. All of the affected syslogs are entering on a custom Input (forwarded from another syslog server) and I would like to keep priority levels 0-4 for processing and siphon the remaining 95% into a black hole.

I have read about blacklisting but can’t seem to find the specific current code or determine and determine the optimal place to put it.

Advice is appreciated. Thank you.

(@_bkeep) #2

If possible, the best solution is not to send the logs in the first place. However, if that is not an option, then you can use the pipeline processor to drop messages.

An example drop rule could look like this

    rule "discard level 0"
    when
        has_field("level") &&
        contains(value: to_string($message.level), search: "0", ignore_case: true)
    then
        drop_message();
    end

Regards,
Brandon