Standalone Node - Upgrade from 5 > 6 - Top_Query Shards Unassigned

[merlin]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I upgraded from Graylog 5 to 6 and everything is working, however I keep seeing unassigned shards that are named top_queries-datehere* , ex. top_queries-2025.02.14-91874

I am in a standalone mode, I believe it’s trying to create replica’s which on all my indices I have set to 0, however this isn’t an index I have created. I remove these shards, however they return after performing queries.

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
{
“index” : “top_queries-2025.02.14-91874”,
“shard” : 0,
“primary” : false,
“current_state” : “unassigned”,
“unassigned_info” : {
“reason” : “INDEX_CREATED”,
“at” : “2025-02-14T14:05:04.475Z”,
“last_allocation_status” : “no_attempt”
},
“can_allocate” : “no”,
“allocate_explanation” : “cannot allocate because allocation is not permitted to any of the nodes”,
“node_allocation_decisions” : [
{
“node_id” : “TeLQrLc8QE2PfPauRctwWg”,
“node_name” : “plasgl”,
“transport_address” : “172.20.1.10:9300”,
“node_attributes” : {
“shard_indexing_pressure_enabled” : “true”
},
“node_decision” : “no”,
“weight_ranking” : 1,
“deciders” : [
{
“decider” : “same_shard”,
“decision” : “NO”,
“explanation” : “a copy of this shard is already allocated to this node [[top_queries-2025.02.14-91874][0], node[TeLQrLc8QE2PfPauRctwWg], [P], s[STARTED], a[id=a-1Cu1QzSaWaQoFTNoCyoQ]]”
}
]
}
]
}

2. Describe your environment:

• OS Information: Ubuntu 22.04.5 LTS

• Package Version:
  graylog-6.1-repository/stable,now 1-1 all [installed]
  graylog-datanode/stable,now 6.1.6-1 amd64 [installed]
  graylog-server/stable,now 6.1.6-1 amd64 [installed]
  mongodb-org/jammy,now 6.0.20 amd64 [installed]
  opensearch/stable,now 2.19.0 amd64 [installed]

3. What steps have you already taken to try and solve the problem?
curl -XDELETE localhost:9200/top_queries-2025.02.13-91873

4. How can the community help?
Anyone else run into this issue after upgrading from 5 > 6 on a standalone node? Thanks!

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[heck-gd]

We’re having the same problem right now. In our case, we had already been running Graylog 6 for months and it seems to be a new issue introduced in Graylog 6.1.6 (or possibly opensearch 2.19?).

[merlin]

Thanks, maybe someone has the answer!

[Wine_Merchant]

This seems to be a feature of Opensearch, in both instances had your Opensearch been updated?

Perhaps dropping replica’s to 0 for the top_queries index set would fix this problem.

[heck-gd]

top_queries is not an index set that can be configured under System/Indices, and apparently it also does not respect the default index set template, which correctly states 0 replicas on our instance.

I could probably deploy an index template directly in Opensearch that forces replicas to 0 for those indices, but it feels like a hacky workaround.

[Wine_Merchant]

System indices created via OS will always need to be managed within OS, Graylog can only affect those indices it manages.

I would suggest trying the below to remove completely, it seems like this is the plugin responsible for the creation of the index.

bin/opensearch-plugin remove query-insights

If you are interested in retaining the top queries index then try something like the below

PUT top_queries*/_settings
{
  "index" : {
	"number_of_replicas" : 0
  }
}

[heck-gd]

Ahh, I didn’t realize that the top_queries feature comes from Opensearch itself. I’ve disabled it now and deleted the indices it created, thanks!

[merlin]

Thanks! Good information!