Delete aws ecs logs from graylog

[kostyanius (Kostya)]

Hello,

i am using AWS ECS with docker containers.
That is why i have faced with an issue that excpet needed logs in Graylog Fluentd is sending also AWS ECS agent logs.
There is no any chance to fixit on Fluentd side so as container log names are changing all the time and i can not fix config Fluentd config file and restart it to send only the proper file.
So only one thing what could be done - delete aws ecs agent logs on Graylog side using pipeline.
I have created one with the next rule:

rule “DropOnText”
when
contains("[INFO]", to_string($message.message)) OR
contains ("[ERROR]", to_string($message.message))
then
drop_message();
end

But it did nothing!Still see all the messeges in search result.
And 0 procceeded messages in output!

Please help to find the root cause and fix it!

Thanks.

[jan (Jan Doberstein)]

He @kostyanius

I would go not direct this dramatic way. Did you have any other ability to check the messages you want to drop?

I would first mark them in one rule to be the one you want to delete and in a second rule - if this mark is present, drop the messages. This would allow you easily to debug the behaviour.

So how does your messages look like you want to drop?

[kostyanius (Kostya)]

Hello @jan
The messages look like (please see attached screen)

[jan (Jan Doberstein)]

sorry that I wasn’t clear on that:

When you open one of the messages - what already extracted fields did you have? I just want to try to find something that will make it less dramatically to select the messages that should be deleted.

[kostyanius (Kostya)]

There is only [INFO] or [ERROR] words which could be used for message deletion. And there is a strict requirement - delete messages where ERROR and INFO only in upper case, so as there are also another logs with “info” and “error” words in application messages (but in lower case) which should not be deleted but processed by another rules (which are not ready for the moment). Thereby, as i can see the message content is quite different and any other words can not be used as selector for deletion.
So, is there any chance to delete those messages? How? Please advise.
Thanks

[kostyanius (Kostya)]

I have tried to simulate message processing and as i can see the pipeline rule doesn`t affect message text. Why?

[jan (Jan Doberstein)]

Is the field “level” accurate? means 1 for Info level and 2 for error? If yes, you can just work with that number and it will be faster than making a regex on the field message.

[kostyanius (Kostya)]

Unfortunately this is not an option so as there is also 1 and 2 level in another log messages which should not be deleted!

[jan (Jan Doberstein)]

so but you could look for the source and the level and that uniq combo can be deleted?

Just to have something that is lighter than fire a regex.

[kostyanius (Kostya)]

Source is the same for both - an internal instance IP address.
So, looks like need to use Regex to solve this task somehow.
Yes?

[kostyanius (Kostya)]

Have managed to find solution:

rule “DropOnText”
when
regex(“[INFO]”, to_string($message.message)).matches == true OR
regex(“[ERROR]”, to_string($message.message)).matches == true
then
drop_message();
end

Can close the topic.

Thanks.