i am using AWS ECS with docker containers.
That is why i have faced with an issue that excpet needed logs in Graylog Fluentd is sending also AWS ECS agent logs.
There is no any chance to fixit on Fluentd side so as container log names are changing all the time and i can not fix config Fluentd config file and restart it to send only the proper file.
So only one thing what could be done - delete aws ecs agent logs on Graylog side using pipeline.
I have created one with the next rule:
rule “DropOnText”
when
contains("[INFO]", to_string($message.message)) OR
contains ("[ERROR]", to_string($message.message))
then
drop_message();
end
But it did nothing!Still see all the messeges in search result.
And 0 procceeded messages in output!
I would go not direct this dramatic way. Did you have any other ability to check the messages you want to drop?
I would first mark them in one rule to be the one you want to delete and in a second rule - if this mark is present, drop the messages. This would allow you easily to debug the behaviour.
So how does your messages look like you want to drop?
When you open one of the messages - what already extracted fields did you have? I just want to try to find something that will make it less dramatically to select the messages that should be deleted.
There is only [INFO] or [ERROR] words which could be used for message deletion. And there is a strict requirement - delete messages where ERROR and INFO only in upper case, so as there are also another logs with “info” and “error” words in application messages (but in lower case) which should not be deleted but processed by another rules (which are not ready for the moment). Thereby, as i can see the message content is quite different and any other words can not be used as selector for deletion.
So, is there any chance to delete those messages? How? Please advise.
Thanks
Is the field “level” accurate? means 1 for Info level and 2 for error? If yes, you can just work with that number and it will be faster than making a regex on the field message.
rule “DropOnText”
when
regex("[INFO]", to_string($message.message)).matches == true OR
regex("[ERROR]", to_string($message.message)).matches == true
then
drop_message();
end