Duplicate logs in dashboard/search screen

[brijesh.kalavadia (Brijesh)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Getting Duplicate logs… we have recently set up Graylog 4.3 multinode environment having one master, master-data and data node. we are seeing duplicate logs in dashboard. we have set up stream for GELF and SYSLOG with having separate index sets and have also selected “Remove matches from ‘All messages’ stream” in stream set up… how would I stop having duplicate logs ?

[gsmith]

Hello && welcome @brijesh.kalavadia

This depends on how you setup this environment.
Normally Dup’s are cause by the log shipper. You have multiple streams with multiple index sets?

That is the only reason why Graylog itself would duplicate messages. When a message is in multiple streams that have different index sets as target.

[brijesh.kalavadia (Brijesh)]

Thanks for prompt reply… yes I have GELF and SYSLOG with having separate index sets… but shouldn’t “Remove matches from ‘All messages’ stream” settings does removes duplicates ???
Or is there a way I can stop having duplicates ?

It will be good to have some workaround in place because having duplicate logs on screen some times annoying users while they are running any search query and looking for some logs…

[gsmith]

Hello,

From what information is shown, I’m not sure if its a Setting/Configuration issue. What I have experienced with Dup’s message was either misconfiguration with Streams/Indices, Log shippers.
Pipelines for work-around might be you best bet, but that just a patch, its not fixing this issue.

Maybe check this out here
Hope that helps

[brijesh.kalavadia (Brijesh)]

I went through the link you provided as well before and in each discussion I found that if you have set up stream store data for separate index set then you will see duplicate data each for stream/index… (Nothing helpful solutions) so I guess I should stop using stream and its settings “Remove matches from ‘All messages’ stream” because that is not working as it suppose to…

My settings are

[gsmith]

Hello,

Here is an Idea, Instead of using gl2_source_input to route message from a INPUT to a INDEX try using something like this.

BTW thanks for the screen shots I was able to see what’s going on.

Example:

I have stream that uses a different index.

Stream Linux Servers

Linux Server Stream Rules configuration BUT I don’t use gl2-souce_input, I use TYPE “match input” as shown in the red boxes.

You can tell the difference from the Result Section.

If that what you trying to accomplish.

[brijesh.kalavadia (Brijesh)]

Yup… That works thanks for your help.