Currently there is only one stream (the default All Messages stream). There is only one node. There is only one index (which does currently have 2 indices, of which only one is marked as active write index).
I have verified the raw logs do not contain duplicates, and I have verified that the above rsyslog config is the only service sending logs.
Any advice on where to continue troubleshooting would be appreciated.
Unfortunately you killed all of my ideas:(
I have only one, what won’t solve it, but it can help to start debugging.
use tcpdump to find where can be the problem, on the server or the client.
tcpdump -i any host CLIEN_IP and port YOUR_GRAYLOG_INPUT -AAAA
it will show the content of the package, and you check the client send twice or the server has a missconfig/bug
Yah, I did a bunch of searching and ruled out all the easy answers I could find, which is really frustrating. I’ll checkout your idea with tcpdump as soon as I can, just have to wait for my sysadmin to be able to do that.
Just an update, I figured out the problem. I had two rsyslog config files (on the same server) sending two different logs to the same Graylog input (I don’t know if the single Graylog input was the problem). Even though the logs were completely different, this for some reason still caused duplication. My fix was to simply combine the two rsyslog config files into one.
Such a commonplace cause to what seemed like a mysterious situation 