Help needed for a pipeline rule

[shenke (Sascha Henke)]

Hi,
I’m asking for a hint, how to write a specific pipeline rule.
We have some legacy systems sending logs into GL and every night around 2 am they do a “logrotate” even with log streaming enabled.
During this rotation they send out a message which “confuses” the existing extractor.

[graylog_6204][1] failed to execute bulk item (index) index {[graylog_deflector][message][a4079f23-5e7c-11ea-8868-005056b9f011], source[{"level":-1,"gl2_remote_ip":"10.10.179.82","gl2_remote_port":60143,"streams":["5b4c781cd8a69b66bcddf65e","000000000000000000000001"],"gl2_message_id":"01E2M39PGKERRFAZKR8R517RWX","source":"10.10.179.82","message":"#Remark: 4812240063 \"proxy12\" \"10.10.179.82\" \"Graylog\"","gl2_source_input":"5b39f908d8a69b11403a5e7a","src_ip":"\"10.10.179.82\"","Kategorie":"4812240063","gl2_source_node":"a0ca8cbc-c3d2-43a4-90d3-111cc34d4358","facility":"Unknown","timestamp":"2020-03-05 01:00:00.913","http_response_code":"\"Graylog\""}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [src_ip] of type [ip] in document with id 'a4079f23-5e7c-11ea-8868-005056b9f011'

The corresponding error message is

Caused by: java.lang.IllegalArgumentException: '"10.10.179.82"' is not an IP string literal.`

For this reason I want to implement a pipeline, which drops these messages since they are not useful for us. Where should I start? I wasn’t able to create this rule only looking at the documentation.

Best regards,
Sascha

[tmacgbay]

If you already have a pipeline going you can have something similar to the rule below in the initial stage to drop the message:

rule "PA-Firewall - ex0 - DROP LDAP config"
when
    // Drop ldap config system messages - we don't currently track queries
    regex(pattern: "(ldap cfg (.*) connected to server)", value: to_string($message.message)).matches == true
then
    drop_message();
end

[shenke (Sascha Henke)]

Cool, thx.
I’m not sure, what to put in the RegEx, but I think, I’ll find identical terms for filtering.

[shoothub]

Hi @shenke
this simple snippet should works for you:

rule “drop remark”
when
regex(pattern: “^#Remark.*”, value: to_string($message.message)).matches == true
then
drop_message();
end