Graylog Cluster is only processing on one node

Graylog Central (peer support)
1

Hi all,

I use a graylog 3-node cluster for my log processing. There are about 500k messages/day getting in.
After switching from logstash to filebeat as log processor, I saw that only one of my nodes is properly processing the messages. The other ones are only enabled, when the master node is completely used by a certain amount of messages. (figured that out after re-enabling big hosts).
I already checked the input, it’s global and other settings do not look very suspicious to me. Do you have any ideas how to fix this?

2018_12_20_15_55_53_Graylog_Nodes
2018_12_20_15_55_53_Graylog_Nodes.png1647×482 54 KB

2

Depends on your setup - either tell filebeat to use all 3 hosts (by adding them all to the hosts array in filebeat.yml) and optionally loadbalance them, or set up a loadbalancer (haproxy or similar) to distribute the load to all nodes.

Messages received on one node are processed on that node, and not sent to another one for processing, as far as I’m aware.

1 Like
3

Already have that … I’m sending my logs to a VIP, where the haproxy behind is distributing the load to all nodes.
The loadbalancing is also configured as: balance leastconn

1 Like
4

That’s odd then, I’d suspect haproxy isn’t aware of the other nodes, or if you have recently added them and changed your setup, depending on whether you reloaded or restarted haproxy, it may have kept the established connections open - filebeat tries to keep it’s connection alive, so that may cause issues as well.

I’m kind of out of ideas here :smiley:

1 Like
5

Oh thanks, thats a good guess I haven’t tried out yet!
It wouldn’t be a huge surprise, if the killing of the haproxy didn’t work properly. I did a re-deploy just recently and it’s possible, that the haproxy wasn’t killed properly. Will try out your solution and give feedback then :slight_smile:

6

It honestly is the only thing I can think of :smiley: If that wasn’t it, we’ll have to see if anyone else has ideas :smiley:

7

filebeat tries to have the connection open as long as possible - that is different to logstash.

that is why you need to handle the connections different. Have all nodes in a RR LB in the filebeat.yml

1 Like
8

Ah okay, thanks for that information, Jan! I just wanted to say, that the restart of the haproxy didn’t help that much. But because we created the Graylog Cluster with a VIP, which is the logstash-destination for filebeat, I think the loadbalance: true option won’t work on this one … maybe I try to change the behaviour of the balance to RR in haproxy and see if that solves the problem …

Thanks for all your answers so far! Will add the solution, if I find it within the next week.

Besides that I wish you a merry christmas :christmas_tree::gift:

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.