I have been working with Graylog for quite some time in my company, we have had a satisfactory and pleasant experience to the point that we have decided to build a more robust infrastructure, as a result we have a Graylog cluster (3 nodes GrayLog 2.5.1, replica set MongoDB v3 .6.10 and Elasticsearch 6.6.0) all mounted on a Red Hat Linux 7.6 OS
We are using Nginx load balancer for the inputs, and this is where I have doubts about the correct configuration.
We have several platforms that are integrated into the graylog with its own inputs and ports in global mode, the inputs are of various types as is the case of syslog udp so far I have no problems as far as I see are working properly, but in the inputs of Beats type
I have noticed that the messages are not being distributed in a balanced way by Nginx where a range of received messages is more than 90% by a single Graylog node
What I am looking for is that the load distribution be balanced in equal proportions, if there is any way of knowing to what extent messages are being received by the nodes.
I put here the configuration of the Nginx that is used to balance this entry
As a last query, as it would work balancing on the same sidecar agent that is currently pointed to the nginx balancer, but we could configure the three graylog nodes there instead of the balancer.
Nginx doesn’t load balance normal TCP connections (such as Beats) at all, only HTTP(S) so what you do is where you have your arrow, you click that - and in Hosts you list the IP and port of every host that runs an input. For example, if you start a global beats input on port 7015, every host in your cluster will have a port 7015 open with a beats input behind. If you then turn your ‘Hosts’ field into ['1.2.3.4:7015','5.6.7.8:7015','12.13.14.15:7015'] and set Load balancing enabled, Filebeat will select a host to connect to at random, and as such you achieve load balancing.
If you need “really real” load balancing, you’ll have to run something like HAProxy to forward connections from the load balance machine to the graylog inputs on the backend servers. Alternatively with a new enough nginx you can achieve the same result using the ‘stream’ stanza.
Thank you very much for the prompt response, the version of nginx that we have is 1.14.2, I will look for documentation if for this version it is possible to do the real balancing as you indicate and in case it does not support it we will try the option of balancing the Collector, I will be commenting the results .
Not totally true.
I use Nginx to loadbalance beats, syslog, syslog over TLS without problem. You should use the config under streams in nginx config. Also nginx can handle UDP, what haproxy can’t.
