I have been trying for a number of hours without success to implement a pipeline rule which drops messages when a field contains an IP address from a specific subnet. I have found a few similar examples in posts but they don’t seem to be having the desired effect.
The graylog input is a sonicwall firewall sending logs via udp syslog. Various extractors are used to parse the traffic, I am trying to drop messages when a field “Int_Src_IP” equals an address in the 172.16.40.0/24 subnet.
Message Filter Chain is before Pipeline Processor in the Message Processors Configuration section.
The two different rules (and some variations) I have tried to use are:
rule “Drop NAT from specific Subnet”
when
contains(to_string($message.Int_Src_IP), “172.16.40.*”, false)
then
drop_message();
end
And:
rule “Drop NAT from specific Subnet”
when
has_field(“Int_Src_IP”) AND $message.Int_Src_IP == “172.16.40.*”
then
drop_message();
end
As additional background re my config, I also have the following pipeline rule in place (working) which drops any messages which dont contain have “NAT Mapping” in the msg field.
rule “Drop all but Rule”
when
NOT contains(to_string($message.msg), “NAT Mapping”, false)
then
drop_message();
end
I have tried these rules in both the same and separate stages but am still seeing the unwanted messages with 172.16.40.0/24 addresses being stored.
I would be very grateful is someone could point me in the direction of where i might be going wrong.
Thanks for your suggestion but neither seem to work.
I’m having some doubts about Graylog picking up the variable i’m using. Is there any documentation which shows how to debug or validate a variable?
Graylog is currently being spammed with log entries from a particular set of devices and due to run out of space before the end of the day so really desperate to get this resolved by filtering out these messages.
you can use the debug() function and watch what it’s output is in the Graylog logs:
tail -f /var/log/graylog-server/server.log
rule "Drop NAT from specific Subnet"
when
starts_with(to_string($message.Int_Src_IP), "172.16.40")
then
debug("Source IP match:");
debug(to_string($message.Int_Src_IP));
drop_message();
end
if nothing shows you can change the when to has_field("Int_Src_IP") just to make sure you get to the debug
Thanks for clarify the debug process, so i have tried with both the starts_with and has_field but my server.log file is almost strangely quite. I don’t believe I have previously made any changes to internal logging config, do I need to enable debugging at all? The last entry’s in this log are from the server starting following a reboot today 5 hours ago.
My log4j2.xml file contains(in case this is relevant):
<?xml version="1.0" encoding="UTF-8"?>
<!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
<Memory name="graylog-internal-logs" bufferSize="500"/>
</Appenders>
<Loggers>
<!-- Application Loggers -->
<Logger name="org.graylog2" level="info"/>
<Logger name="com.github.joschi.jadconfig" level="warn"/>
<!-- This emits a harmless warning for ActiveDirectory every time which we can't work around :( -->
<Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
<!-- Prevent DEBUG message about Lucene Expressions not found. -->
<Logger name="org.elasticsearch.script" level="warn"/>
<!-- Disable messages from the version check -->
<Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
<!-- Silence chatty natty -->
<Logger name="com.joestelmach.natty.Parser" level="warn"/>
<!-- Silence Kafka log chatter -->
<Logger name="kafka.log.Log" level="warn"/>
<Logger name="kafka.log.OffsetIndex" level="warn"/>
<!-- Silence useless session validation messages -->
<Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
<Root level="warn">
<AppenderRef ref="rolling-file"/>
<AppenderRef ref="graylog-internal-logs"/>
</Root>
</Loggers>
You shouldn’t need to turn anything on. did you try changing the when part to make sure you can look at what is coming in? Be sure to comment out the drop_message() since it may catch more than what you want to drop. Or create/apply a new temp rule:
rule "Drop NAT from specific Subnet-TEST"
when
has_field("Int_Src_IP")
then
let debug_message = concat("Source IP match-test: ", to_string($message.Int_Src_IP));
debug(debug_message);
end
It works! Thank you so much for your help I couldn’t have done it without it.
So I was using the following debug rule in stage 1, but was still getting no output in the server.log file:
rule “Debug Field”
when
has_field(“Int_Src_IP”)
then
let debug_message = concat(“Debug-Out:”, to_string($message.Int_Src_IP));
debug(debug_message);
end
But had this rule in stage 0:
rule “Drop all but Rule”
when
NOT contains(to_string($message.msg), “NAT Mapping”, false)
&&
NOT contains(to_string($message.msg), “Assigned IP”, false)
&&
NOT contains(to_string($message.msg), “DHCP”, false)
then
drop_message();
end
The issue (which I still don’t really understand) seemed to be the order. As a test I applied the debug rule (using a different/relevant field) to a different stream which had no other pipeline rules configured, this resulted in the expected debug output being generated in server.log.
Returning to the pipeline rule I was having issues with I removed the stage 0 rule and placed it in stage 1, then applied the debug rule at stage 0. This then generated the debug output in server.log.
I was then able to replace the debug rule at stage 0 with the following which is working at desired:
rule “Drop NAT from Listed Subnets”
when
contains(to_string($message.Int_Src_IP), “172.16.40”, false)
OR
contains(to_string($message.Int_Src_IP), “172.16.50”, false)
OR
contains(to_string($message.Int_Src_IP), “172.16.60”, false)
then
drop_message();
end
