A large number of duplicate logs

Hi all,
I meet some problems. Yesterday, a server that graylog collected logs was restarted. When this server is started, the contents of the log files on this server are all re-entered into the graylog. By the way, Collector-sidecar is configured for boot-on self-starting. Then I searched for a log in graylog and found two duplicate results.


This log is just a record in the log file, but there are two results found in graylog. They entered through different graylog nodes and had different IDs.
I am confused. Is it because the restart-caused collector-sidecar thinks that the contents of this log file need to be re-collected?
I would appreciate it if you help me.

The log entry is in two different indices. As you can see the uuid of the messages are different.

I do not know why your collector shipped the messages a second time, the registry of filebeat might be corrupted or something changed the inodes of the files. So you need to check the filebeat logs, by default in /var/lib/graylog-sidecar/collectors/filebeat/log to see the reason for that.

Thank you!
I checked /var/log/graylog/collector-sidecar/filebeat file. The contents of the file at that time are as follows:


Do you have any ideas?

no idea - because the log messages does not show any information.

Sherlock, you need to make a deep dive into your system. Check filebeat logs from the past. The registry of filebeat and find the reason why this information is lost.

That is nothing that just “happens” or that is very common. It must be something that is uniq to your system and you need to find that - because nobody that has no access to the system can’t find that.

OK, thank you so much! Currently I have checked the filebeat log file without exception. I will look at something else.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.