Elasticsearch cluster graylog is yellow with 3 unassigned opensearch configs and plugins?

[grayl0g-is-not-easy]

So I spun up the newest version of graylog for my house on:

$ uname -a
Linux graylog 5.10.0-24-amd64 #1 SMP Debian 5.10.179-5 (2023-08-08) x86_64 GNU/Linux

Everything is local, nothing special or fancy. It’s been working fine for about 3 months now. Recently I checked the status and found my Elasticsearch cluster graylog is yellow . I do not know when or why this happened, so I started to educate myself and found this information after digging around:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED


  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2835  100  2835    0     0  74605      0 --:--:-- --:--:-- --:--:-- 74605
.opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED CLUSTER_RECOVERED
.plugins-ml-config                        0 r UNASSIGNED CLUSTER_RECOVERED
.opensearch-sap-log-types-config          0 r UNASSIGNED CLUSTER_RECOVERED

Ummm ok, what the heck does that mean besides I have 3 indexes that are unassigned. Doesn’t really tell me why…soooo I dug some more.

curl -XGET 'localhost:9200/_cluster/allocation/explain'


{"index":".opensearch-sap-pre-packaged-rules-config","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2023-08-17T17:28:41.278Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"vs8XWHBRSqCucxyMWhn2RA","node_name":"graylog","transport_address":"192.168.10.105:9300","node_attributes":{"shard_indexing_pressure_enabled":"true"},"node_decision":"no","deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[.opensearch-sap-pre-packaged-rules-config][0], node[vs8XWHBRSqCucxyMWhn2RA], [P], s[STARTED], a[id=s9bULvoCSmm-RmGxJnrDRA]]"}]}]}

OK - don’t really care. Maybe it was because I rolled back a vm snapshot, maybe it was an update. No clue how this happened but trust me, it was not always there. Maybe it was related to the hypervisor input I added that seemed to have too many messages and flood the system. So I deleted the hypervisor input, index, everything about it. That did not solve the problem, so, I decided to just delete the errors above after a lot more reading.

curl -XDELETE http://localhost:9200/.opensearch-sap-pre-packaged-rules-config
curl -XDELETE http://localhost:9200/.plugins-ml-config
curl -XDELETE http://localhost:9200/.opensearch-sap-log-types-config

Then I checked again to make sure they were really gone (and they were)

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

Rebooted the server and, ummm yeah, they instantly came back. HELP!

Joy, this platform just said “new users can only post 1 pic” well…instead of many pictures showing all the details, you get one

[drewmiranda-gl (Drew Miranda)]

This is an issue with opensearch and unfortuantely them choosing to set the default replica shard value when creating indexes to 1. This means if you have only a single node opensearch cluster, those replica shards have nowhere to go.

We’ve opened up a couple of issues with opensearch but have not really gotten much engagement or even acknowledgement about the issue. To be fair the issue only exists for single node opensearch clusters but they also clearly state single node clusters are supported so there is a bit of disconnect there.

all of this to say, there isn’t much we can do to address this. The only thing you can do is to manually change the replica count on any offending indexes AFTER they have been created (the command only applies to existing indexes):

Replace the target address to suit your needs and replace .opendistro with the start of the index or indexes you want to update.

curl -X PUT "http://127.0.0.1:9200/.opendistro-*/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'

Hope this helps.

[grayl0g-is-not-easy]

Did the following, then rebooted.

$ curl -X PUT "http://127.0.0.1:9200/.opensearch-sap-pre-packaged-rules-config/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'
{"acknowledged":true}
$ curl -X PUT "http://127.0.0.1:9200/.plugins-ml-config/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'
{"acknowledged":true}
$ curl -X PUT "http://127.0.0.1:9200/.opensearch-sap-log-types-config/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'
{"acknowledged":true}
$ curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2475  100  2475    0     0   302k      0 --:--:-- --:--:-- --:--:--  345k

Guess what…It’s GREEN AGAIN! Thanks good sir!. And wow that sucks that they won’t fix this.