Could not execute search when using searches over 14 days

Graylog Central (peer support)
1

Recently I have started seeing the following errors when I do all kinds of different searches in Graylog. Most of the time shortening the time period I am searching fixes it. This is the error I get.

Error Message:
Unable to perform search query.
Details:
Search status code:
500
Search response:
cannot GET https://graylog.company.com:443/api/search/universal/relative?query=%28devname%3300D_Primary%20OR%20devname%3A300D_Secondary%29%20AND%20msg%3A%22SSL%20tunnel%20established%22&range=1209600&limit=150&sort=timestamp%3Adesc (500)
2

Searching through my ES logs I see the following line every time I do a search.

Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@3b977b7d on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4498364f[Running, pool size = 10, active threads = 10, queued tasks = 994, completed tasks = 15376]]

3

There are too many indexing and search requests for your Elasticsearch cluster.

Either increase the task queue sizes on all Elasticsearch nodes (which doesn’t really solve the problem) or make sure that the Elasticsearch cluster has enough resources to cope with the message throughput.

4

I am only bringing in around 150 messages a second so I don’t have a huge load on this system. I increased the core count from 6 to 10 but it still isn’t able to keep up.

How do I increase the search queue size on the appliance?

5

That’s not possible in the OVA. Manual changes to the configuration of Graylog, Elasticsearch, or MongoDB will be overwritten when running graylog-ctl reconfigure.

Try checking the logs of Elasticsearch for errors and warnings: http://docs.graylog.org/en/2.3/pages/configuration/file_location.html#omnibus-package

6

Alright, it is finally time to move to a dedicated system then. Here are my thoughts on what I need to do.

  1. Spin up a real Graylog instance (Not appliance)
  2. Add to current cluster
  3. Make sure replica = 1
  4. Let everything sync
  5. Decommission old graylog appliance

Am I missing anything?

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.