Hello,
I have two Graylog nodes that are behind a load balancer. Yesterday, I noticed the In msg/s were twice the Out msg/s.
I looked at the nodes and the second Graylog node was showing the following:
The journal contains 12,089,442 unprocessed messages in 50 segments. 1,239 messages appended, 627 messages read in the last second.
The first Graylog node showed:
The journal contains 8 unprocessed messages in 1 segment. 4 messages appended, 4 messages read in the last second.
I asked our load balancer admin and they advised the stats of packets going to each Graylog node were pretty equal. Then last night the load shifted from the second Graylog server to the first.
They are both showing
Current lifecycle state:Running
Message processing:Enabled
Load balancer indication:ALIVE
The only error I see in both nodes logs are from our load balancer.
2018-09-12T15:37:27.274-04:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a1fc9d12-b6ab-11e8-ac91-00505693bfd3, journalOffset=203760632, codec=syslog, payloadSize=19, timestamp=2018-09-12T16:48:10.815Z, remoteAddress=X.X.X.X.
As I am typing this on the first (overworked) Graylog node:
Current lifecycle state:Throttled
Message processing:Enabled
Load balancer indication:THROTTLED
Thanks,