Having a hell of a time getting a cluster working

Graylog Central (peer support)
1

Hey folks, having a hell of a time getting a cluster working. I have 3 servers, each running Elasticsearch, Mongo and Graylog. I have the Mongo replica set working ( at least it appears to work as the master can connect to it) and the Elasticsearch cluster also works.

I can log into to the master (log01) and can do most things like create inputs. This is where things get a little strange.

First, I can see logs coming in and if I look at ES using hq i can see lots of documents

Index # Docs Primary Size # Shards # Replicas Status
traps_0 0 159.0B 1 0 open
graylog_0 8,951,677 1.8GB 1 0 open
firewall_2 0 636.0B 4 0 open
firewall_1 6,760,827 1.4GB 1 0 open
firewall_0 1,980,091 477.0MB 1 0 open

However the search screen shows the nothing found message.

That’s when I noticed these messages in the log file for the master

2017-03-23T17:39:38.819Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:38.822Z WARN  [ProxiedResource] Unable to call http://10.1.240.17:12900/system on node <80534f52-9aab-492e-b119-41f5dc752d1b>, result: Internal Server Error
2017-03-23T17:39:39.884Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system/metrics/multiple on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:41.884Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system/metrics/multiple on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:43.814Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:43.817Z WARN  [ProxiedResource] Unable to call http://10.1.240.17:12900/system on node <80534f52-9aab-492e-b119-41f5dc752d1b>, result: Internal Server Error
2017-03-23T17:39:43.886Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system/metrics/multiple on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:45.884Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system/metrics/multiple on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:47.888Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system/metrics/multiple on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:48.814Z WARN  [ProxiedResource] Unable to call http://10.1.240.16:12900/system on node <0a27a76c-f2fa-4dab-908f-5cf80d216150>, result: Not Found
2017-03-23T17:39:48.818Z WARN  [ProxiedResource] Unable to call http://10.1.240.17:12900/system on node <80534f52-9aab-492e-b119-41f5dc752d1b>, result: Internal Server Error

I’ve spent the better part of my day trying to resolve this. and I’ve tried various “solutions” that I found on Google but none work.

Graylog version: Graylog 2.2.2+691b4b7

log01 config
rest_listen_uri = http://0.0.0.0:12900/api/
rest_transport_uri = http://10.1.240.15:12900/api/
web_listen_uri = http://0.0.0.0:9000/
elasticsearch_network_host =10.1.240.15
elasticsearch_network_bind_host = 10.1.240.15
elasticsearch_discovery_zen_ping_unicast_hosts = 10.1.240.16:9300, 10.1.240.17:9300


log02 config
rest_listen_uri = http://0.0.0.0:12900/api/
rest_transport_uri = http://10.1.240.16:12900/api/


log03
rest_listen_uri = http://0.0.0.0:12900/api/
rest_transport_uri = http://10.1.240.17:12900/api/

Any help would be extremely useful. As a follow up. Is it worth it to do a cluster since 99% of the logs are from routers and firewalls? After spending all this time I feel like having two separate all in one servers is the way to go.

2

Hey @eyeball,

try removing the /api/ from your rest_listen_uri config lines.
Look at the log, it is trying to connect to http://10.1.240.16:12900/, not http://10.1.240.16:12900/api/.
You only need the /api/ at the end, if you have the rest_listen and the web_listen running on the same port.

Greetings - Phil

1 Like