No Route to Host - Graylog not Starting

Graylog Central (peer support)
1

After restart graylog nodes, they cannot communicate with the elasticsearch database. Graylog version is 2.2.3, ElasticSearch is 2.4.4. Graylog logs report the following:

2017-04-19T09:24:30.906-04:00 INFO  [zen] [graylog-c27e0f18-5abd-453d-93f3-57be2ffa5b11] failed to send join request to master [{001-elasrchdb-02.example.com}{iT4HNsk3Rza4ifz9CfHxIg}{172.16.1.154}{172.16.1.154:9300}{attr.zone=prod, attr.rack_id=even}], reason [RemoteTransportException[[001-elasrchdb-02.example.com][172.16.1.154:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-c27e0f18-5abd-453d-93f3-57be2ffa5b11][172.16.1.61:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[no_route_to_host_exception: No route to host]; ]

If I do a netstat on the elasticsearch nodes, I see connections and doing a tcpdump -A reports similar to the following:

."=..$...client.true.data.false.master.false..|...........001,graylog-c27e0f18-5abd-453d-93f3-57be2ffa5b11.bmujzCKZSNS9rarcTWFGeA.172.16.1.61.172.16.1.61...

so it is making to the host.

2

All Elasticsearch nodes must be able to connect back to the embedded ES node running inside Graylog.

In your case, the Elasticsearch node must be able to connect to 172.16.1.61:9350.