[type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized

[momodiouf (Diouf)]

Hello, I’m new to graylog and I installed elasticsearch which works very well when I do a systemctl status elasticsearch

i am on ubuntu 20.04 and i am using Gralog 4.2.7

when i go to my graylog dashboard i have this message : “Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]”

when i show my graylog logs i see this output :

2022-03-24T08:34:32.002Z ERROR [IndexerClusterCheckerThread] Uncaught exception in periodical
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:93) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.indicesExist(ClusterAdapterES7.java:290) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.clusterHealth(ClusterAdapterES7.java:269) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.health(ClusterAdapterES7.java:82) ~[?:?]
at org.graylog2.indexer.cluster.Cluster.health(Cluster.java:70) ~[graylog.jar:?]
at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:60) ~[graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [http://127.0.0.1:9200], URI [/graylog_,gl-events_,gl-system-events_?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 503 Service Unavailable]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1866) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.lambda$indicesExist$13(ClusterAdapterES7.java:290) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 13 more
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [HEAD], host [http://127.0.0.1:9200], URI [/graylog_,gl-events_,gl-system-events_?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 503 Service Unavailable]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974) ~[?:?]
at org.graylog.storage.elasticsearch7.ClusterAdapterES7.lambda$indicesExist$13(ClusterAdapterES7.java:290) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 13 more

thanks for your help

[gsmith]

Hello && welcome

I might be able to help. I need to ask for more information about this environment. First I would like to make sure the configuration are correct.

Can you show your Elasticsearch && Graylog config file?

Graylog Configuration file, this will remove extra lines in the file that is not needed to look at.

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

Elasticsearch Configuration file.

cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"

Next,

• Do you have firewall or Selinux/Apparmor enabled?
• I assume this is a Package installation?
• Elasticsearch /MongoDb and Graylog on the same node?
• What versions are each service, Elasticsearch/MongoDb/Graylog?
• Last question, what have you done so far to resolve this issue?

Thanks

[momodiouf (Diouf)]

Hello thank you for help i solve the problem by add a cluster uuid.

[gsmith]

Hello @momodiouf

Glad you fixed the issue,
Can I ask, this is kind of odd that you had to add a cluster UUID.

When the ES yaml file was configured it would have generated the UUID for you.
Example:

#give your cluster a name.
cluster.name: graylog

#give your nodes a name (change node number from node to node).
node.name: "es-node-1"

#define node 1 as master-eligible:
node.master: true

#define nodes 2 and 3 as data nodes:
node.data: true

#enter the private IP and port of your node:
network.host: 172.11.61.27
http.port: 9200

Output:

[root@graylog graylog-server]# curl http://127.0.0.1:9200
{
  "name" : "es-node-1",
  "cluster_name" : "graylog",
  "cluster_uuid" : "OMgi3eu5QGiJ3buKOYn4_w",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

By chance was this a clone or something?

NOTE:
The elasticsearch-node unsafe-bootstrap tool will generate a new cluster UUID.
If you could mark this as resolve that would be great for future searches.
Thanks

[momodiouf (Diouf)]

Hello @gsmith

no not at all it’s not a clone, I still don’t understand why it didn’t generate me uuid when installing elasticsearch. the cluster uuid value was _na.

Thank you for your help

i marked this as resolve