Exclude all streams other than All messages by query

[chawkimatta]

Hello,
I have a message log that is routed to multiple streams without deleting it from “All messages” in order to perform alerts and some other analytics filtering…

The issue is that when we search for the log w get it duplicated, for ex the log below is showing 3 times because its available in 3 streams (i have more than 10 stream)

*is there a way to exclude all streams other than “All messages”? to get unique logs ?
*can we check what are the streams we have after performing a query in “All messages” in order to exclude them ?

Thank you,

[macko003]

I started from the other side, and I think you will get your answer.
So If you store one message in multiple streams, with different index sets, the GL will store the message in multiple times. So if you search, you got all messages.
Use one index set for your 3 different streams.
OR Start the search from streams way, and you will see only one message when you start to search.

[chawkimatta]

Thank you @macko003 for your reply
you are right,
both critical streams shown in the snapshot, are on a index with prefix “critical_alarms”
and “All messages” stream is on another index with prefix “graylog”
what we can do here is excluding the index starting with “critical_alarms” while searching “All messages”

will try that…

[macko003]

Honestly, I don’t have a solution for that.
Don’t put one massage to different index sets. Config to use all streams the same index set.

[chawkimatta]

@macko003

Don’t put one massage to different

@macko003 can you elaborate more please ?
is it to spare disk space?
avoid duplication?
or other considerations…
please advise,

[macko003]

@macko003

So If you store one message in multiple streams, with different index sets, the GL will store the message in multiple times.

I can just repeat myself.
Check your streams, what index sets its using.

[chawkimatta]

@macko003 we have 2 messages Stored in index: critial_alarms_2
and 1 message stored in index “graylog_537”

when i add AND !_index:critial_alarms_2 to the query i get the correct list.
but here we want to exclude any critical_alarms index…

AND !_index:critial_alarms_*  is not working ...

[jan (Jan Doberstein)]

He @chawkimatta

you would select only the specific streams you want to search in and do not use the only for admins possible search across all streams. This will allow to get only the results that are inside this stream and the index set that is the base for this stream.

[chawkimatta]

@jan

you would select only the specific streams you want to search in and do not use the only for admins possible search across all streams. This will allow to get only the results that are inside this stream and the index set that is the base for this stream.

Hi @jan
Acctually we need to have all messages in one place and we are using the other streams for alerts filtering.
so the main debugging stream is the “All messages”…
that would be great if !index:critial_alarms* works…

[cawfehman]

Not sure which you created first, but this seems to be a duplicate thread

Exclude a specific Index from search

Hello, I have message logs stored on 2 indexes with prefix graylog & critial_alarms NB: critial_alarms is for critical detections stream. graylog is “All messages” stream. [image] [image] the search is showing duplication of logs… how can we exclude index critial_alarms* from “All messages” search? (critial_alarms_1,critial_alarms_2,critial_alarms_3…) !_index:critial_alarms_2is working fine while !index:critial_alarms* is not… Please advise,

try not to duplicate