Unable to perform search query: unknown field [disable_coord]

[lvicknair (Lauren Vicknair)]

Description of your problem

Trying to upgrade to a version that is compatible with IRSA (above 7.9) and search queries are breaking with a error around “unknown field [disable_coord]”. I saw their was a previous post on this but no answer.

Description of steps you’ve taken to attempt to solve the issue

I have tried deploying different versions of elasticsearch to try to get around the issue - 7.7, 7.8, 7.9, 7.10

Operating system information

docker
k8s

Package versions

• Graylog - 4.1.3
• MongoDB -
• Elasticsearch - trying to upgrde this
• Curator - 5.8.4

[gsmith]

Hello && Welcome

Need to ask a couple question. When you see this error on the Web UI what does Graylog log files show? is there anything that would pertain to this issue? Did you check Elasticsearch files?
What was your procedure of upgrading Elasticsearch?
Did you have to roll back elasticsearch ? If so, how did you go about doing it (i.e. ES snapshot, Virtual machine Checkpoint, etc…)?

EDIT: What’s the output of the following command’s if you execute it on the machine running Graylog?

curl -XGET http://localhost:9200/_cluster/health?pretty=true

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

[lvicknair (Lauren Vicknair)]

Thanks for responding!

We are using terraform in combination with helm to manage our graylog. Steps include:

• Pausing message processing on nodes
• Bumping the ES docker image
• Apply using terraform which then
• Kicks off a helm release
• After release is complete, manually killing all 3 master nodes one by one so they restart
• Unpause processing

Rollbacks process incliudes:
*Pausing message processing

• Scaling all stateful sets replicas to 0
• Deleting all Persistent Volume Claims for each stateful set
• Terraform apply the old working version of ES
• Upload snapshot

In the process of upgrading to 7.9.3 to get you the curl data

[lvicknair (Lauren Vicknair)]

Upgraded to 7.10.1 instead (helm compatibility)

[elasticsearch@dev-graylog-elasticsearch-master-0 ~]$ curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "dev-graylog-elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 10,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 261,
  "active_shards" : 522,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[elasticsearch@dev-graylog-elasticsearch-master-0 ~]$ curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}

[gsmith]

Hello,

Unfortunately I haven’t had the opportunity to use Terraform & Helm yet. I might be much help in this situation.

I see you execute ES snapshot. By chance are you using a custom template for ES?

EDIT: I’m unsure if it will solve your issue but you might try rotating the index/indices against which your widgets and searches querying.