Duplicate logs/messages

Hi all,

i just started using graylog and have actually a basic installation with only one node which is running fine so far. When I created a new index set to separate FW logs (another retention etc.) I started to receiving messages twice (new fw index and default index). My stream, related to the firewall logs, are pointing to the fw index and “Remove matches from ‘All messages’ stream” is enabled. But all messages are stored in fw index as well in the graylog index (default).
There is no other stream where the fw logs are routed by rule and point to the default index, except of the “all messages” stream.

I am currently using
Graylog 3.0.2
Elasticsearch 5.6.16
MongoDB 3.2.7

Does anyone know how to fix this error?

if you “all messages” stream contain the same messages as the “fw stream” and both streams have different indices as storage, that is the reason for this.

Hi Jan, thanks for the fast reply…how i can solve the problem? To have the fw logs only in the specific fw indices?

remove the messages from the default stream - that is the solution.

Hi Jan,

how can i remove the messages from the default stream? Thought this will happen automatically, with enable “Remove matches from ‘All messages’ stream” on the FW stream.

br
Robert

This will/should happen. But if you have a stream that does match this message too with the location to store in the default index you will have double saved messages.

Did you checked if you have other stream rules that match this messages?

Yes i did several times. I see the logs in the FW logs stream with information like stored in index “firewall_2” routet into streams “All_messages, FW_logs”. This is how is should be. When i use the full_message of the log and searching for it under the All messages stream i find the same message just with the information stored in index “graylog_3” routet into streams “All_messages, FW_logs”