Dashboard Error - Unable to perform search query: OpenSearch exception

Graylog Central (peer support)
,
1

Hello Community,
I’m reaching out to the community as my last resort. Even though I’ve been researching and found similar community posts, I couldn’t find a solution to my issue.

1. Describe your incident:
Everything has been working fine until I have logged back in after 3 days or so.
I’m encountering an issue with my Graylog setup where I’m unable to view some dashboards. About 95% of them are giving me the following error:

While retrieving data for this widget, the following error(s) occurred:

Unable to perform search query: OpenSearch exception [type=too_many_nested_clauses, reason=Query contains too many nested clauses; maxClauseCount is set to 1024].

And occasionally I am getting this notification:

Aggregation search failed:
OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed].

2. Describe your environment:

  • OS Information: Docker; Host - Ubuntu,
  • Package Versions:
    • Graylog: 5.1.5
    • OpenSearch: 2.9.0
    • Lucene: 9.7.0

3. What steps have you already taken to try and solve the problem?
I’ve already tried several troubleshooting steps without success, including:

  • Verifying cluster health (it’s currently green).
  • Checking node status (one node, no issues).
  • Reducing Grok patterns.
  • Deleting some of the dashboards.
  • Checking disk space (there’s no issue with space).
  • Community research

Cluster:

{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 168,
  "active_shards" : 168,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Additionally, all shards are “green” when checking :9200/_cluster/health/?level=shards&pretty .

Nodes:

 "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "docker-cluster",
  "nodes" : {
    "0Jb8m0qZS0epq3hdUlh89g" : {
      "name" : "5dfc5fa46db4",
      "transport_address" : "172.x.x.x:9300",
      "host" : "172.x.x.x",
      "ip" : "172.x.x.x",
      "version" : "2.9.0",
      "build_type" : "tar",
      "build_hash" : "1164221ee2b8ba3560f0ff492309867beea28433",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 33,
        "mlockall" : false
      }
    }
  }
}

Despite my efforts and research, the issue persists.

4. How can the community help?
I’m reaching out to the community for assistance in resolving this issue. If anyone has insights into why I’m getting the error, I would greatly appreciate your guidance and any suggestions.

Additionally, I’ve attached a screenshot of indexes:

indexes
indexes1226×1326 72.6 KB

Thank you in advance for your help!

2

Do any of your dashboard widgets contain free text searches that don’t limit to a single field? Ie “some text” rather than field: text.

I’ve seen those cause that error. Also a single widget getting the error will cause all to display that error, so you will need to check them all.

1 Like
3

Thank you for your suggestion! I took a look at all the dashboards and reviewed the searches, and the issue was exactly like you mentioned, and that was indeed causing the error. Once I have edited those searches everything started working just fine.
Thanks again for your help!

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.