Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
I am running a rather large Graylog 6.3.3 cluster with 12 data nodes and 12 server nodes.
Over the weekend, several of the data nodes disks filled to 100% and stopped working.
I have added more space for these nodes and Graylog is back up and running.
But, the OpenSearch cluster datanode-cluster is showing red and displaying that there are 1152 unassigned shards.
“OpenSearch cluster datanode-cluster is red. Shards: 3095 active, 0 initializing, 0 relocating, 1152 unassigned”
Since the data-nodes are configured to use certificates for node communications, how to I execute the curl commands to the OpenSearch data-nodes to delete the unassigned shards?
For example, the curl command below to view the unassigned shards returns “curl: (52) Empty reply from server”
curl -XGET “http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason”
If I need to use the Graylog API to remove the unassigned shards, which one do I use, and what is the syntax?
2. Describe your environment:
-
OS Information: Rocky Linux 9.6
-
Package Version:Graylog 6.3.3+700dd8f
-
Service logs, configurations, and environment variables:Logs are large, which log segments would be helpful?
3. What steps have you already taken to try and solve the problem?
I have attempted several different ways to delete these shards. What worked in Elasticsearch in prior Graylog major versions does not work with the OpenSearch data-nodes.
4. How can the community help?
I need help with the commands/API that will delete these unassigned shards.
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]