Good evening. I’ve got a busy input - GELF UDP. Services sending logs for this input set remote host as an A record for the IPs of the three Graylog server nodes in the cluster I have inherited. A couple of days ago I changed this input from ‘Global input’ to run on one of three Graylog server nodes. Later on that day I changed it back to Global. Now looking back I see, during the period where the input was set to one Graylog server, messages missing from services that log via this input. These are services that are in constant use and log messages all the time but for which there are zero messages recorded during the hours where the input was not global.
I’m trying to explain the missing messages.
It occurs to me that I did not make a change to the A record to remove the two IPs for the Graylog servers that did not have this input running. As this is UDP we don’t have much to trace from the owners of the services that are missing messages. There is nothing much in the Graylog server logs. Detail for the single node that was running the input continued to record incoming and outgoing messages all through the time where it was the only node taking the input.
In addition when I changed the input back to global the input didn’t appear active on the other two Graylog nodes. I needed to sudo graylog-ctl restart to make that happen. Is that expected or should we be able to switch between without a restart?
Thanks very much.
Graylog version 2.3.2