Graylog cluster load balancing problem

Hello community,

I am installing a graylog cluster using the following architecture.

My software versions:
CentOS 8
graylog 4.0.5
mongo 4.4.4
elastic 7.10.1
nginx 1.14.1

My configurations:
GRAYLOG1
ip: 192.168.1.8
hostname: graylog01

config:
is_master = true
http_bind_address = 192.168.1.8:9000
http_publish_uri = http://$http_bind_address/
rest_listen_uri = http://192.168.1.8:9000/api
http_external_uri = http://graylog.mydomain.net/
trusted_proxies = 192.168.1.12/32

GRAYLOG2
ip: 192.168.1.15
hostname: graylog02

config:
is_master = false
http_bind_address = 192.168.1.15:9000
http_publish_uri = http://$http_bind_address/
rest_listen_uri = http://192.168.1.15:9000/api/
http_external_uri = http://graylog.mydomain.net/
trusted_proxies = 192.168.1.12/32

GRAYLOG3
ip: 192.168.1.16
hostname: graylog03

config:
is_master = false
http_bind_address = 192.168.1.16:9000
http_publish_uri = http://$http_bind_address/
rest_listen_uri = http://192.168.1.16:9000/api/
http_external_uri = http://graylog.mydomain.net/
trusted_proxies = 192.168.1.12/32

nginx
ip: 192.168.1.12
hostname: graylog

config:
stream {

    upstream cluster_syslog {
             server 192.168.1.8:514;
             server 192.168.1.15:514;
             server 192.168.1.16:514;
    }
    server {
            listen 192.168.1.12:514 udp;
            proxy_timeout 10s;
            proxy_pass cluster_syslog;
    }

}

http {
upstream cluster_http {
server 192.168.1.8:9000 max_fails=3 fail_timeout=30s;
server 192.168.1.15:9000 max_fails=3 fail_timeout=30s;
server 192.168.1.16:9000 max_fails=3 fail_timeout=30s;
}

    server {
            listen 192.168.1.12:80;
            server_name graylog.mydomain.net;

            location /
            {
                    proxy_set_header    Host $http_host;
                    proxy_set_header X-Forwarded-Host $host;
                    proxy_set_header X-Forwarded-Server $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Graylog-Server-URL http://$server_name/;
                    proxy_pass http://cluster_http;
            }
    }

}

Http requests to the graylog cluster or syslog messages are reaching the servers correctly.
The problem is in accessing the api.
When I access http://graylog.mydomain.net/system/nodes it identifies the 3 nodes but with the information
“System information is currently unavailable.”
Acessing http://graylog.mydomain.net/system/overview identifies the cluster ID as well as the indication that it has 3 active members.

On the servers I have the following errors (on each member)
WARN [ProxiedResource] Unable to call http://api/system/metrics/multiple on node : api: Name or service not known
WARN [ProxiedResource] Unable to call http://api/system/metrics/multiple on node : api

Yes, without a FQDN or IP.

I have been searching but until now all attempts have always resulted in this problem.
Does anyone have a tip to point in the right direction?

Thank you all so much for your time.

@FMPS
Hello,

Maybe I can help with your problem.
First thing I noticed was this.

The "rest_listen_uri = " I remember seeing that in Graylog2 configuration file, but since then configuration files has change. Looks like you have configuration from two different versions of Graylog.

This is mine, I have Graylog 4.0.6, but I’m running TCP/TLS.

http_bind_address = ipaddress:9000
http_publish_uri = https://graylog.localhost.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://ipaddress:9200

Here is Graylog 4 server.conf

https://docs.graylog.org/en/4.0/pages/configuration/server.conf.html

Here is Graylog 2 server.conf

https://docs.graylog.org/en/2.4/pages/configuration/server.conf.html

So that might be an issue.

2 Likes

Hello @gsmith ,

You are right.

I removed “rest_listen_uri” and made following changes:
http_bind_address = 0.0.0.0:9000
http_publish_uri = http://192.168.1.8:9000/
http_external_uri = http://graylog.mydomain.net/

Everything is working just fine :slight_smile:

Thank you very much for your time.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.