ES2
GL1
GL2
GL3
Links removed.
What happened to the logs of the other Elasticsearch node?
The other node was cleanly shut down, then destroyed. Long story short, we were seeing kernel IOWAIT on that VM and our VMware group wanted to replace it with a copy of a clean node that was not exhibiting those issues (namely, a clone of the secondary ES node). That is being completed today.
Any ideas why Graylog would not fail over to the other Elasticsearch node?
Anyone? Having working Elasticsearch failover is pretty important to us in this environment, and if it cannot fail over properly even on a clean shutdown of an Elasticsearch node that’s a big issue.
My guess would be that your elasticsearch cluster needs to have a minimum of 3 nodes and the number of required master eligible nodes of 2 in order to the ES to properly survive a node failure.
But this would be on the ES side. I don’t know if this is a problem (was the surviving ES node the master node of the cluster).
EDIT: I mean; to have the fault tolerance in the ES cluster, you would set all three ES nodes as master-eligible node.master: true, and the discovery.zen.minimum_master_nodes to 2 (that is (n/2)+1) https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html
Both ES nodes are master-eligible, and the current setting for the cluster is 1 for minimum master. One node was shutdown, and the other node almost immediately assumed the master role as expected. However, Graylog continually attempted to contact the node that was down, essentially ignoring the new master node. Nothing that I have in the logs suggests that Graylog was unable to contact or write to the node that newly assumed the master role, just that Graylog could not find the node that was down.
For the UDP packet drops, what I did to solve them eventually - I pulled all the system buffers up to 50MB via sysctl.conf and also adjusted all the UDP buffers on each of the UDP inputs to 10MB. Thus far, I’ve not had any more UDP drops on any of the Graylog nodes.
This is one of those instances where Telegraf/InfluxDB/Grafana can really help diagnose cluster health and reveal things lurking under the hood you may not otherwise realize is happening.
@jochen @jtkarvo what would you recommend for solving the Elasticsearch failover issue? Only thing I could do at this point is attempt to spin up some master-only instances on the Graylog nodes themselves. Would this be a big deal? Do we really think it will solve anything given the other node already did assume the master role properly?
hi,
unfortunately, I don’t know if that would help. I don’t have enough knowledge to answer this.
I would add a VM as Master Only Node but not spin that up on the Graylog Nodes - because than you have Graylog and Elasticsearch fighting for the resources.
Happy birthday @jan!
@jochen it appears I spoke too soon regarding the UDP buffer drops - yesterday I noticed that the master node had to re-register a bunch of times, and I stumbled on the fact that all of my Graylog nodes were swapping to disk. I throttled the heap back to 1G instead of 2G, and low and behold today I have a ton of drops again when the CPU is under stress.
Is there some kind of indication or guidance as to how large the Graylog heap has to be?
EDIT: Looks like all my UDP plugin buffers got reset to 1MB when the nodes were restarted yesterday. Does this setting not save on the inputs?
EDIT2: Setting udp_recvbuffer_sizes in the server.conf apparently overrides all the input buffer settings on startup (i.e. any changes to input buffers made in the GUI will be lost on reboot of that node).
would you mind open a bug report for that please