How to solve .opendistro_security and .plugins-ml-config in UNASSIGNED state

Graylog Central (peer support)
1

Hi all, since my graylog datanode disk full, I had add some new nodes, and my cluster has enough disk space now. But my Open Search cluster still in yellow state, like this:

OpenSearch cluster datanode-cluster is yellow. Shards: 3985 active, 0 initializing, 0 relocating, 8 unassigned

I had checkd the unassigned shades, got this:

[root@graylog-data1 ~]# curl -k -XGET “https://1.1.1.1:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason” --cert /root/cert/client.crt --key /root/cert/client.key --cacert /root/cert/ca.crt | grep UNAS
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0.opendistro_security 0 r UNASSIGNED REPLICA_ADDED
.opendistro_security 0 r UNASSIGNED REPLICA_ADDED
.opendistro_security 0 r UNASSIGNED NODE_LEFT
.opendistro_security 0 r UNASSIGNED REPLICA_ADDED
100 218k 100 218k 0 0 1086k 0 --:–:-- --:–:-- --:–:-- 1086k
.plugins-ml-config 0 r UNASSIGNED REPLICA_ADDED
.plugins-ml-config 0 r UNASSIGNED REPLICA_ADDED
.plugins-ml-config 0 r UNASSIGNED NODE_LEFT
.plugins-ml-config 0 r UNASSIGNED REPLICA_ADDED

Seems two shards, .plugins-ml-config and .opendistro_security are in UNASSIGNED state. But my cluster had enoungh disk space, maybe I should do somthing to the two shards?

So, how can I solve this to make my cluster to green? Thanks

2

Hello @baalkchina,

For shards with NODE_LEFT you can force reallocation with the below.

curl -XPOST "https://localhost:9200/_cluster/reroute?retry_failed=true&pretty"  -u admin:password --insecure

For the REPLICA_ADDED shards, you have the power the reduce replica count of system shards which is probably for the best here.

curl -XPUT "https://localhost:9200/.plugins-ml-config/_settings?pretty" -u admin:password --insecure -H 'Content-Type: application/json' -d '{
  "index": { "number_of_replicas": 0 }
}'