Getting message “Could not execute search”

Graylog Central (peer support)
1

Hello,

We used graylog 3.1.4 + elasticsearch 6.8.13 + mongodb 4.2 during 1 month wihtout problems but since few days, when I launch search I encounter this following error message :

# Could not execute search

There was an error executing your search. Please check your Graylog server logs for more information.

Error Message:

Unable to perform search query

Details:

Search status code:

500

Search response:

cannot GET http://X.X.X.X:9000/api/search/universal/relative?query=*&range=0&filter=streams%3A602a71281153c00202023123&limit=150&sort=timestamp%3Adesc (500)

I have already checked space on disk just used at 14%
All Services elasticsearch mongod and graylog-server runs perfectly
No failed indexing attempts in the last 24 hours.Show errors

But

Collection containing a total of 161,324 indexer failures.

5 days ago graylog_0 65bf5a08-778f-11eb-b3de-00505694f6ca {“type”:“unavailable_shards_exception”,“reason”:"[graylog_0][1] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_0][1]] containing [132] requests]"}

Could help to me to fix this issue please

Thank you very much for your time and your help

Regards

2

@sysadm
What’s in the logs of your Graylog and Elasticsearch node/s?

3

looks like you lost an ES node… or are having an issue with a shard. By default, Graylog builds indices with 4 primary shards and 0 replicas… you’ll need all those shards up to run queries… sounds like you have a Elasticsearch issue.

what is Graylog reporting for a status on your ES?

or run

curl -X GET "localhost:9200/_cluster/health?pretty"
1 Like
4

Hello,

Tahnk you for your feedback, please see below result of following command curl -X GET “localhost:9200/_cluster/health?pretty”

curl -X GET “localhost:9200/_cluster/health?pretty”
{
“cluster_name” : “graylog”,
“status” : “red”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 22,
“active_shards” : 22,
“relocating_shards” : 0,
“initializing_shards” : 4,
“unassigned_shards” : 2,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 2,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 21175,
“active_shards_percent_as_number” : 78.57142857142857
}

5

@sysadm
Hello,
As @cawfehman mentioned you have some issues with your Elasticsearch.

You can try using Elasticsearch’s cat shards API it might tell you which shards are unassigned.
Each row lists the name of the index, the shard number, whether it is a primary p or replica r shard, and the reason it is unassigned

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

You can also use the cluster allocation explain API to try to garner more information about shard allocation issues

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

Maybe this might help.

2 Likes
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.