I recently upgraded from Graylog 3.3, Elasticsearch 6.8 and MongoDB 3.x and am now running Graylog 4.0.1, Elasticsearch 7.10.2 and MongoDB 4.0.22. For the most part, things seem to be fine. My searches and dashboards are working, no indexing errors, I can rotate indexes, etc.
However, when I rotate an index, I get this error in the server.log:
2021-01-15T18:01:38.277-06:00 ERROR [SystemJobManager] Unhandled error while running SystemJob [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:122) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:92) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:85) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.optimizeIndex(IndicesAdapterES7.java:394) ~[?:?]
at org.graylog2.indexer.indices.Indices.optimizeIndex(Indices.java:326) ~[graylog.jar:?]
at org.graylog2.indexer.indices.jobs.OptimizeIndexJob.execute(OptimizeIndexJob.java:71) ~[graylog.jar:?]
at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:834) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.forcemerge(IndicesClient.java:897) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$optimizeIndex$22(IndicesAdapterES7.java:394) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:90) ~[?:?]
… 13 more
Caused by: java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
at org.graylog.shaded.elasticsearch7.org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[?:?]
… 1 more
I still see the index optimize system job in the web interface and it appears to run and complete, but I’m just a little concerned about the exceptions in the log. Is this an actual problem that I should take some steps to remedy?
Thanks,
Mark