Graylog Nodes "Out of Balance"

Graylog Central (peer support)
1

Hello,
I have two Graylog nodes that are behind a load balancer. Yesterday, I noticed the In msg/s were twice the Out msg/s.

I looked at the nodes and the second Graylog node was showing the following:
The journal contains 12,089,442 unprocessed messages in 50 segments. 1,239 messages appended, 627 messages read in the last second.

The first Graylog node showed:

The journal contains 8 unprocessed messages in 1 segment. 4 messages appended, 4 messages read in the last second.

I asked our load balancer admin and they advised the stats of packets going to each Graylog node were pretty equal. Then last night the load shifted from the second Graylog server to the first.

They are both showing
Current lifecycle state:Running

Message processing:Enabled

Load balancer indication:ALIVE

The only error I see in both nodes logs are from our load balancer.

2018-09-12T15:37:27.274-04:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=a1fc9d12-b6ab-11e8-ac91-00505693bfd3, journalOffset=203760632, codec=syslog, payloadSize=19, timestamp=2018-09-12T16:48:10.815Z, remoteAddress=X.X.X.X.

As I am typing this on the first (overworked) Graylog node:

Current lifecycle state:Throttled

Message processing:Enabled

Load balancer indication:THROTTLED

Thanks,

2

he @burd

what kind of input did you use? Some Inputs are better loadbalanced than others. The Logshipper you use might have also issues with that.

Without knowing the above we can only :crystal_ball:

3

I am only using a syslog udp input.

  • allow_override_date:false

  • bind_address:0.0.0.0

  • expand_structured_data:true

  • force_rdns:true

  • override_source:<empty>

  • port:1514

  • recv_buffer_size:1048576

  • store_full_message:true

4

how does your LB actually balance? Maybe it does not balance well - if you see lots of the mentioned error message you might want to try a raw input and then just extract all information yourself. Just be able to accept everything and not loose anything.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.