I have such errors in ES logs (and my dashboards don’t show up). ES 5.5.2, Graylog 2.3.1, about 30T of data in 12 ES nodes.
org.elasticsearch.transport.RemoteTransportException: [es-graylog-node04][ES-node-IP:9300][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@652844fa on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5c799098[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 1634784]]
What is you suggestion - more memory to ES nodes, or something else?
Increasing threadpool sounds like not solving the issue, or is the Graylog way of using ES such that it would be beneficial?