Graylog elasticsearch health red, how to fix?

[tatuh]

We are running Graylog in kubernetes along with elasticsearch. Recently we had issue with diskspace, and elasticsearch cluster started showing status red. Just added more diskspace, and tried to restart the pods(elasticsearch) in cluster, but ended up in the situation where pods/elasticsearch app wont start as the logs show cluster status is red.

So I think the main issue is how to recover elasticsearch cluster from status red?

This issue is about elasticsearch, but I believe graylog users are facing this, so I’ll try to look for the fix also from here…

curl -X GET “localhost:9200/_cluster/health?pretty”
{
“cluster_name” : “elasticsearch”,
“status” : “red”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 3,
“active_primary_shards” : 120,
“active_shards” : 120,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 2,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 98.36065573770492
}

[aaronsachs]

You have unassigned shards. Have you looked at the Elasticsearch explain API documentation and tried to see if that gives you any useful information?

[tatuh]

Thank you for clarifying. I was able to detect the unassigned shrads, and then successfully rerouted the unassigned shards. After this Elastic cluster red status was solved, and I was able to use Graylog again.
…I had an issue with one indicie which was in read only state, recalculating the index ranges from graylog ui.