Drop logs with pipeline rules

[csorzi]

Hello guys!

I am trying to drop HTTPD logs which we don’t need, with a pipeline. (graylog 2.5)
I have a Syslog UDP input with extractors, one of them extracts the IP where the connection comes from into an “ipv4_address” field.
So we have a lot of APIs and they are called from a lot of servers and producing a load of unnecessary logs, but these connections are coming from the same subnet, lets call it 10.10.10.0/24. I dont wanna drop all the api logs, because we need some of them(calls from different subnet)

So I tried to create a rule like this:

rule "drop from internal subnet"
when
    cidr_match("10.10.10.0/24", to_ip($message.ipv4_address))
then
    drop_message();
end

It isn’t working. To be sure I created a second rule to check if the messsage has the extracted ipv4_address field.

rule "has ipv4 fields"
when
    has_field("ipv4_address")
then
end

So now my pipeline is like:

pipeline "Drop internal subnet httpd logs"
stage 1 match all
   rule "has ipv4 fields"
   rule "drop from internal subnet"
end

Still no luck, what I am doing wrong?

Thanks in advance!

[Ponet (Jesse Hills)]

In your first rule that checks whether the ipv4_address field exists, you could add a new field to test what the returning result of the to_ip function is (maybe there is something going wrong there?)

I’d try something like set_field("Parsed_IPv4",to_ip($message.ipv4_address))

If there is some issue with the above, you could also try converting back into a string…
set_field("Parsed_IPv4",to_string(to_ip($message.ipv4_address)))

[benvanstaveren]

IIRC the rules in a stage aren’t ran in the order they appear in the rule list, if I were to test it, I’d set up 2 stages, first stage with the has ipv4 field check, second stage with the drop rule - then see if your 2nd stage gets any throughput - if it does, it means the ipv4 field exists.

If it still doesn’t work after, make sure the ipv4 field contains an actual IP, no spaces added, no port numbers, etc. to ensure to_ip can handle it. If it still doesn’t work after checking that, then I’m out of ideas

[jan (Jan Doberstein)]

combine both into one rule to be 100% sure:

rule "drop from internal subnet"
when
    has_field("ipv4_address") AND
    cidr_match("10.10.10.0/24", to_ip($message.ipv4_address))
then
    drop_message();
end

[csorzi]

Nice, this helped a bit.

When I did a simulation for the pipeline, i realized the ipv4 address isn’t extracted.
I realized the order is wrong with message processors, Pipeline processing was before Message Filter Chain, so it couldn’t work with the extracted fields.

But then again in one stage it wouldn’t work, so I split it in two stages and now every log I wanted gets dropped

[benvanstaveren]

Excellent! Good to hear Also, keep in mind what @jan said about combining things into a single rule - it’s more efficient that way

[macko003]

and in general always a good idea to check the field exists before use it as a source in conditions or set something.