1. Describe your incident:
Unassigned shards after datanode disconnect. Built the cluster from the “Ubuntu Installation Multiple Graylog Nodes” guide on graylog.org
I’ve resolved that issue but now i’ve got a constantly red cluster.
2. Describe your environment:
-
OS Information:
- Ubuntu 24 LTS
- 3x graylog-servers 4x graylog-datanodes
-
Package Version:
- graylog node v6.3.3+700dd8f, Noir
- graylog data node v6.3.2+667aca0
-
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I believe this a self managed open search. I know why the data node disconnected, but I need to remove those shards to get the cluster out of the red. From the CLI of a datanode I cannot curl -XGET or curl -DELETE. I get errors like “curl: (52) Empty reply from server, or “curl: (7) Failed to connect to nodename port 9200 (or 9000) after 0 ms: Coulden’t connect to server.
I can use the webbrowser and go to http ://nodename:9000/api/any/opensearch/ and after the opensearch/ I can cluster/allocation/explain, or cat/nodes?v=true?v=true&h=name,host,disk.used,disk.total and other commands. I know know what needs to be deleted, I looked in the GUI index sets and I see all graylog_xxx except the 37 missing ones. So I’m at a loss how to remove them, they are old so losing that data is okay. Just want to return the cluster to green.
Until this the Graylog node has been working amazing.
The missing index show a state of unassigned and reason node_left.
4. How can the community help?
Any help on how to delete the shards on this configuration of Gray Log would be amazing.
Thank you.
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]