Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [service_unavailable/1/state not recovered / initialized];]

Graylog Central (peer support)
, ,
1

While retrieving data for this widget, the following error(s) occurred:

  • Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];].

1. Describe your incident:
After installing graylog, I faced a problem
→ curl -XGET ‘http://192.168.180.180:9200/_cluster/health?pretty=true’

image
image714×270 77 KB 

[root@localhost ~]# curl -XGET 'http://192.168.180.180:9200/_cluster/health?pretty=true'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

2. Describe your environment:

  • OS Information:
    centos-release-7-9.2009.1.el7.centos.x86_64
    [elasticsearch-7.x]
    mongodb-org/4.2/x86_64/
    CentOS installation - Installing Graylog

  • Service logs, configurations, and environment variables:
    tail -f /var/log/elasticsearch/graylog.log

org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:11:25,662][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}]; discovery will continue using [0.0.0.0:9300] from hosts providers and [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-13T16:11:28,290][WARN ][r.suppressed             ] [node-1] path: /_alias/graylog_deflector, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=true, ignore_throttled=false, name=graylog_deflector}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:11:33,632][WARN ][r.suppressed             ] [node-1] path: /_alias/graylog_deflector, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=true, ignore_throttled=false, name=graylog_deflector}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:11:35,666][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}]; discovery will continue using [0.0.0.0:9300] from hosts providers and [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-13T16:11:45,668][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}]; discovery will continue using [0.0.0.0:9300] from hosts providers and [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-13T16:11:53,768][WARN ][r.suppressed             ] [node-1] path: /graylog_*,gl-events_*,gl-failures_*,gl-system-events_*, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=false, ignore_throttled=false, index=graylog_*,gl-events_*,gl-failures_*,gl-system-events_*}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:11:55,676][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}]; discovery will continue using [0.0.0.0:9300] from hosts providers and [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-13T16:11:58,302][WARN ][r.suppressed             ] [node-1] path: /_alias/graylog_deflector, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=true, ignore_throttled=false, name=graylog_deflector}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:11:59,395][WARN ][r.suppressed             ] [node-1] path: /_alias/gl-failures_deflector, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=true, ignore_throttled=false, name=gl-failures_deflector}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:12:03,640][WARN ][r.suppressed             ] [node-1] path: /_alias/gl-events_deflector, params: {ignore_unavailable=false, expand_wildcards=open,closed, allow_no_indices=true, ignore_throttled=false, name=gl-events_deflector}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-06-13T16:12:05,678][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}]; discovery will continue using [0.0.0.0:9300] from hosts providers and [{node-1}{lr4yDKFFQV6Mxi6Xi7664A}{yCU3QWc3R5a2VgAF7iUSew}{192.168.180.180}{192.168.180.180:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
2 
# Use a descriptive name for your cluster:
#
cluster.name: graylog
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.

# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.180.180
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts: [0.0.0.0]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
action.auto_create_index: false
3

Hello && Welcome @ductc

First of all " Holy Cow" this was hard to read. I fixed you post for you. Please use the markdown or use this post for a better understanding on the HowTo.

This helps us , help you

As for this…

Maybe try this in ES yaml file.

[root@graylog home]#   cat /etc/elasticsearch/elasticsearch.yml  | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.180.180
http.port: 9200
action.auto_create_index: false
discovery.type: single-node

Restart elasticsearch service.

1 Like
4

Thanks for those instructions. I appreciate it.
I changed the information and the problem was solved.

image
image1050×391 100 KB

Thank you so much!

1 Like
5

No problem, glad i can help. If you can mark this as resolved for future search . That would be great.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.