# Elasticsearch service down
**URL:** https://community.graylog.org/t/elasticsearch-service-down/22290
**Category:** Graylog Central (peer support)
**Created:** [January 11, 2022, 1:59am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290 "2022-01-11T01:59:35Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 1:59am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/1 "2022-01-11T01:59:35Z")
Sometimes in graylog inputs are not getting processed when i checked elasticsearch service is stopped.when i start elasticsearch everything is back to normal.This happens frequently.
Message in indexers failure and logs:
`> graylog_13 d4881fa2-7200-11ec-9125-005056923c4a {"type":"unavailable_shards_exception","reason":"[graylog_13][1] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_13][1]] containing [110] requests]"}`
Pls help
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 2:38am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/2 "2022-01-11T02:38:31Z")
Hello
> [@groovyghost](#):
>
> primary shard is not active Timeout
Are you running out of disk space?
---
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 2:48am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/3 "2022-01-11T02:48:46Z")
no, i have over 500gb free on each node
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 2:51am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/4 "2022-01-11T02:51:09Z")
Ok,
Can you show the output of this command, You may need to adjust it for your environment
`curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty`
---
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 3:05am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/5 "2022-01-11T03:05:31Z")
Please find the output
```auto
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
},
"status" : 400
}
```
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 3:08am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/6 "2022-01-11T03:08:22Z")
Ok,
Next troubleshooting tip . What is the output of this command?
`curl -XGET http://localhost:9200/_cluster/health?pretty=true`
You mention something about…
> [@groovyghost](#):
>
> 500gb free on each node
do you have two elasticsearch nodes?
---
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 3:15am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/7 "2022-01-11T03:15:48Z")
pls find the ouptut
```auto
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 72,
"active_shards" : 72,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
```
do you have two elasticsearch nodes?
3 es-nodes
i.e 3 servers with graylog,mongod,elasticsearch in all of them
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 3:31am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/8 "2022-01-11T03:31:55Z")
Ok,
What I know now is…
- So you have plenty of free space in your /data directory on ALL ES nodes, Correct?
- Cluster is green ALL ES nodes, and I see you have all three nodes shown in that last command.
- No problem with shards.
Just curious did you execute those command on each Elasticsearch node?
Next, troubleshooting tip.
This command will make sure all index are green.
EDIT: I forgot to put this command here 😃
`curl -XGET 'http://localhost:9200/_cluster/health?level=indices&pretty'`
Check list
- Did you make sure elasticsearch service on all nodes were enabled?
`systemctl enable elasticsearch`
- Permission are good on all nodes?
- Have you tried manually rotating you indices to see if this happens again?
This issue sometimes happens when…
1. You running out of disk space.
2. Have to many shard allocated.
3. Out of memory conditions resulted in orphaned Elasticsearch indices \<— I’m leaning toward this is what’s happening.
---
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 3:44am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/9 "2022-01-11T03:44:56Z")
> [@gsmith](#):
>
> So you have plenty of free space in your /data directory on ALL ES nodes, Correct?
yes
glog - 555G
glog2 - 527G
glog3 - 538G
> [@gsmith](#):
>
> Just curious did you execute those command on each Elasticsearch node?
yes and All nodes have same output
> [@gsmith](#):
>
> - Did you make sure elasticsearch service on all nodes were enabled?
> - Permission are good on all nodes?
yeah all are enabled and permissions are good
> [@gsmith](#):
>
> - Have you tried manually rotating you indices to see if this happens again?
No when i start elasticsearch service everything is back to normal
with multiple indexer failures
---
### Author:  [@groovyghost](https://community.graylog.org/u/groovyghost)
#### Post date: [January 11, 2022, 3:49am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/10 "2022-01-11T03:49:01Z")
> [@gsmith](#):
>
> - Out of memory conditions resulted in orphaned Elasticsearch indices \<— I’m leaning toward this is what’s happening.
Does this mean i have to increase JVM heap space?
All the inputs are processed by only one node i need to add loadbalancer but still is 200msgs/sec too much to process just for one node?
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 3:49am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/11 "2022-01-11T03:49:04Z")
Ok,
So since you have a cluster (3) and was this the master node you restart the service on? If so, then you this may transferred a new master. If you havent already this will check all your indices
` curl -XGET 'http://localhost:9200/_cat/indices?pretty'`
---
### Author:  [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2022, 3:51am UTC](https://community.graylog.org/t/elasticsearch-service-down/22290/12 "2022-01-11T03:51:47Z")
> [@groovyghost](#):
>
> Does this mean i have to increase JVM heap space?
This means your starving elasticsearch for memory.
I have no idea what you environment looks like so I can tell you how to fix it unless you share more information. Please look here for a better understanding.
> [@How to Post a Question in the Community that Gets Responses](https://community.graylog.org/t/how-to-post-a-question-in-the-community-that-gets-responses/20879#details-you-should-always-includedetails-11):
>
> This platform is made with love for community discussions on the open source tool Graylog, it components and usage. Here’s a Graylog support-inspired template (thank you, @aaronsachs ) that’ll get responses: Description of your problem \