Cannot connect to ElasticSearch Cluster

Graylog Central (peer support)
1

Trying to set up my cluster and I cannot get the servers to connect.
The Web interface keeps telling me it can’t connect to the cluster.

/etc/graylog-server/server.conf:
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxxxxxxxxxxxxxx
root_username = xxxxxxxx
root_password_sha2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.10.xx.xx:9000/api/
web_listen_uri = http://10.10.xx.xx:9000

rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
#elasticsearch_index_prefix = graylog
#elasticsearch_template_name = graylog-internal
allow_leading_wildcard_searches = false
allow_highlighting = false
#elasticsearch_cluster_name = graylog
#elasticsearch_node_name_prefix = graylog-
#elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.example.com:9300,graylog2.example.com:9300
#elasticsearch_discovery_zen_ping_unicast_hosts = 198.51.100.23:9300, 198.51.100.42:9300
#elasticsearch_node_master = false
#elasticsearch_node_data = false
elasticsearch_network_host = 10.10.xx.xx
#elasticsearch_network_bind_host = 10.10.xx.xx
#elasticsearch_network_publish_host = 10.10.xx.xx
elasticsearch_analyzer = standard

Am I supposed to configure the cluster settings in server.conf or elasticsearch.yml?

I have tried both and a combination of the two, and it still won’t connect.
I’m thinking it may have something to do with the default clustername that was assigned at installation.

Using Curl on the localhost on both machines I get this:
{
“name” : “Kingpin”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “uPaq5N9rS3CmYi7DoXHamw”,
“version” : {
“number” : “2.4.4”,
“build_hash” : “fcbb46dfd45562a9cf00c604b30849a6dec6b017”,
“build_timestamp” : “2017-01-03T11:33:16Z”,
“build_snapshot” : false,
“lucene_version” : “5.5.2”
},
“tagline” : “You Know, for Search”
}

{
“name” : “Master Pandemonium”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “cxNgUKvrRPGFweAxVBe-0w”,
“version” : {
“number” : “2.4.4”,
“build_hash” : “fcbb46dfd45562a9cf00c604b30849a6dec6b017”,
“build_timestamp” : “2017-01-03T11:33:16Z”,
“build_snapshot” : false,
“lucene_version” : “5.5.2”
},
“tagline” : “You Know, for Search”
}

Thanks in advance for any help.

2

try changing
elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.example.com:9300,graylog2.example.com:9300 to the correct hosts and remove the elastsearch_network_host line.

also it looks like the connection to mongodb isn’t present in your config? I suspect you can’t login to the web interface in that case.

lastly you have two seperate clusters there, the elasticsearch.yml will sort that out, until then you will only be able to use one server.

3

What’s the exact error message and what are the logs of your Graylog node?

4

try changing
elasticsearch_discovery_zen_ping_unicast_hosts = graylog1.example.com:9300,graylog2.example.com:9300 to the correct hosts and remove the elastsearch_network_host line.

They are set to the correct host names, I put example.com as a placeholder for posting here.

also it looks like the connection to mongodb isn’t present in your config? I suspect you can’t login to the web interface in that case.

It is, I just didn’t think to include it in my paste. I can access the web UI.

lastly you have two seperate clusters there, the elasticsearch.yml will sort that out, until then you will only be able to use one server.

So am I to use the elasticsearch.yml and NOT server.conf for the elasticsearch configuration?

5

What’s the exact error message and what are the logs of your Graylog node?

2017-03-14T13:01:52.008-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:02:04.327-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2017-03-14T13:02:22.010-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:02:52.012-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:03:04.327-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2017-03-14T13:03:22.015-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:03:52.019-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:04:02.077-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-03-14T13:04:04.328-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2017-03-14T13:04:22.021-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:04:52.023-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-03-14T13:05:04.328-04:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2017-03-14T13:05:22.025-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

Sorry for the late replies. I’ve been out of town for two days.

6

That’s the wrong ES cluster name. Either modify the cluster name in the configuration of your ES nodes (cluster.name) or in the configuration of your Graylog nodes (elasticsearch_cluster_name).

7

Would it be better to modify the ES .yml or modify the graylog-server server.conf? Or does that make any difference?

8

It doesn’t matter which setting you modify as long as the cluster name is identical everywhere.

9

Just want to say, I really appreciate the help.

So, I’ve set elasticsearch_cluster_name in server.conf on both servers, restarted the machines, checked the ES server through curl and I’m still getting it as cluster: elasticsearch. I’m also not using elasticsearch.yml.

Could it be that it’s seeing an existing ES cluster from when I first installed it with the defaults? If so, how do I go about removing the old cluster so a new one can be made?

Edit: I think I may have found the problem. I’m going through the ElasticSearch log and it says:

publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}

If I’m wanting multiple nodes to talk to see each other, they need to be bound to their hostname/IP address, no?

Edit2: I changed the elasticsearch.yml and it now recognises the cluster.

Edit3: SUCCESS! Turns out that the config needs to be set in both elasticsearch.yml and server.conf. The cluster/nodes are up and the Web UI can see them.

Again, thanks for the help!