We’re using UDP gelf as input over three nodes balanced by keepalived. Messages aren’t chunked since the average size is less than 1K.
The nodes have identical resources and there are no special differences between them that would indicate issues.
My problem is that the nodes are being used uneven, with one getting 10k messages input while at the same time another one is getting 600 or a thousand. When marking any one of them as DEAD for the balancer the overall message count falls by one third. For example, we have 6000 incoming messages on three nodes and after removing one we get 4000 messages in, not 6k.
Again, the only input is over a balancer which is using direct routing and round robin.
Any advice or hints are welcome.