How to send the same message several time without duplicate it

Graylog Central (peer support)
1

Hi,

Actually when I send a message with filebeat into elasticsearch, i can define a fingerprint that allow to not duplicate the message if i send it several time.

I would like to send the same message with rabbitmq and use graylog to consume it.
But each time the message is get from the input, graylog add gl2_message_id that is different for the same message

Is there a way to not duplicate the same message into graylog ?

thanks

2

I think that i must use pipeline rule to drop message if the fingerprint value already exist.

But i’he no idea how to do it

3

Wouldn’t it be more efficient for Rabbitmq to deduplicate?

4

In fact the question is:

Does graylog allow to force de _id to avoid duplicate messages ?

With logstash i can force the document_id with a fingerprint function to be sure that i don’t duplicate a message

With graylog is there a way to do a similar operation ?

The aim is to be able to send the same message without to be duplicated into ES.

5

such is not possible with Graylog. At the time of writing this.

you might want to open a feature request for that:

6

I found lot of issue on this question.
I have replied on an existing issue : https://github.com/Graylog2/graylog2-server/issues/466
Hope my suggestion will help

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.