Multiple Errors Between Graylog & Elasticsearch

monarch684 · January 12, 2022, 2:47pm

We have been having some issues lately with our Graylog being able to search logs. When looking at the respective log files in Graylog and Elasticsearch, I see errors but I am not sure how to troubleshoot.

Graylog logs:

2022-01-12T08:26:39.646-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:26:44.787-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:26:44.789-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:26:44.789-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.
2022-01-12T08:27:09.659-06:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-01-12T08:27:09.719-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.721-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:09.721-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.762-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:09.778-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:27:09.778-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:27:09.780-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:27:09.780-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.
2022-01-12T08:27:09.785-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.787-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:09.787-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.804-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:27:09.805-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:27:09.805-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:27:09.805-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.

monarch684 · January 12, 2022, 2:48pm

Logs Continued

2022-01-12T08:27:09.809-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.810-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:09.810-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.887-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:27:09.888-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:27:09.889-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:27:09.889-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.
2022-01-12T08:27:09.893-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:09.895-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:09.895-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:10.309-06:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-163 [ACTIVE]]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-163 [ACTIVE]];, errorDetails=}, retrying (attempt #3).
2022-01-12T08:27:19.537-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:27:19.551-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:27:19.552-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:27:19.552-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.
2022-01-12T08:27:19.610-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:19.611-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:19.611-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index

monarch684 · January 12, 2022, 2:49pm

Logs continued

java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:27:22.390-06:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-172 [ACTIVE]]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-172 [ACTIVE]];, errorDetails=}, retrying (attempt #3).
2022-01-12T08:27:31.400-06:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-171 [ACTIVE]]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-171 [ACTIVE]];, errorDetails=}, retrying (attempt #4).
2022-01-12T08:27:31.401-06:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-160 [ACTIVE]]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-160 [ACTIVE]];, errorDetails=}, retrying (attempt #4).
2022-01-12T08:27:31.513-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:32.497-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:40.587-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:27:44.595-06:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: timeout. - Socket closed.
2022-01-12T08:27:46.187-06:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-01-12T08:27:59.598-06:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: timeout. - Read timed out.
2022-01-12T08:27:59.692-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:28:05.112-06:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-01-12T08:28:14.601-06:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: timeout. - Socket closed.
2022-01-12T08:28:19.596-06:00 ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-176 [ACTIVE]]; nested: SocketTimeoutException[60,000 milliseconds timeout on connection http-outgoing-176 [ACTIVE]];, errorDetails=}, retrying (attempt #4).
2022-01-12T08:28:21.526-06:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-01-12T08:28:21.527-06:00 INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2022-01-12T08:28:21.528-06:00 INFO [MongoIndexSet] Cycling from to <gl-failures_0>.
2022-01-12T08:28:21.528-06:00 INFO [MongoIndexSet] Creating target index <gl-failures_0>.
2022-01-12T08:28:21.546-06:00 WARN [Indices] Couldn’t create index gl-failures_0. Error: Unable to create index gl-failures_0
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to create index gl-failures_0
at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
… 16 more
Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://10.3.208.43:9200], URI [/gl-failures_0?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{“error”:{“root_cause”:[{“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”}],“type”:“validation_exception”,“reason”:“Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;”},“status”:400}
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.create(IndicesClient.java:145) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$create$3(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
at org.graylog.storage.elasticsearch7.IndicesAdapterES7.create(IndicesAdapterES7.java:153) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:196) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:28:21.547-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-01-12T08:28:21.547-06:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_312]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-01-12T08:28:21.559-06:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.

monarch684 · January 12, 2022, 2:50pm

Elascticsearch logs

[2022-01-12T08:21:32,195][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430919][73127] duration [7.2s], collections [1]/[17.2s], total [7.2s]/[23.8h], memory [5.1gb]->[4.9gb]/[7.7gb], all_pools {[young] [0b]->[84mb]/[0b]}{[old] [4.7gb]->[4.7gb]/[7.7gb]}{[survivor] [14mb]->[52mb]/[0b]}
[2022-01-12T08:21:32,195][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430919] overhead, spent [7.2s] collecting in the last [17.2s]
[2022-01-12T08:21:37,345][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430920][73128] duration [4.1s], collections [1]/[5.1s], total [4.1s]/[23.8h], memory [4.9gb]->[4.8gb]/[7.7gb], all_pools {[young] [84mb]->[0b]/[0b]}{[old] [4.7gb]->[4.8gb]/[7.7gb]}{[survivor] [52mb]->[38mb]/[0b]}
[2022-01-12T08:21:37,355][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430920] overhead, spent [4.1s] collecting in the last [5.1s]
[2022-01-12T08:21:47,563][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430923][73129] duration [7.2s], collections [1]/[8.2s], total [7.2s]/[23.8h], memory [5.1gb]->[4.8gb]/[7.7gb], all_pools {[young] [264mb]->[0b]/[0b]}{[old] [4.8gb]->[4.8gb]/[7.7gb]}{[survivor] [38mb]->[5.7mb]/[0b]}
[2022-01-12T08:21:47,563][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430923] overhead, spent [7.2s] collecting in the last [8.2s]
[2022-01-12T08:21:47,563][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/lnx_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43776}] took [7216ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:21:47,571][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43778}] took [7216ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:22:15,426][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430949][73135] duration [2.5s], collections [1]/[2.8s], total [2.5s]/[23.8h], memory [5.1gb]->[4.8gb]/[7.7gb], all_pools {[young] [272mb]->[0b]/[0b]}{[old] [4.8gb]->[4.8gb]/[7.7gb]}{[survivor] [30.8mb]->[26.1mb]/[0b]}
[2022-01-12T08:22:15,426][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430949] overhead, spent [2.5s] collecting in the last [2.8s]
[2022-01-12T08:22:33,172][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430960][73137] duration [7.4s], collections [1]/[7.7s], total [7.4s]/[23.8h], memory [5gb]->[4.9gb]/[7.7gb], all_pools {[young] [148mb]->[0b]/[0b]}{[old] [4.8gb]->[4.8gb]/[7.7gb]}{[survivor] [43.9mb]->[52mb]/[0b]}
[2022-01-12T08:22:33,173][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430960] overhead, spent [7.4s] collecting in the last [7.7s]
[2022-01-12T08:22:39,783][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430964][73138] duration [2.8s], collections [1]/[3.6s], total [2.8s]/[23.8h], memory [5.2gb]->[4.9gb]/[7.7gb], all_pools {[young] [308mb]->[0b]/[0b]}{[old] [4.8gb]->[4.8gb]/[7.7gb]}{[survivor] [52mb]->[28mb]/[0b]}
[2022-01-12T08:22:39,783][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430964] overhead, spent [2.8s] collecting in the last [3.6s]
[2022-01-12T08:23:08,588][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/_msearch?typed_keys=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43786}] took [5450ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:23:08,650][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430988][73140] duration [5.3s], collections [1]/[5.8s], total [5.3s]/[23.8h], memory [5.1gb]->[5.1gb]/[7.7gb], all_pools {[young] [264mb]->[156mb]/[0b]}{[old] [4.8gb]->[4.9gb]/[7.7gb]}{[survivor] [41.5mb]->[39.7mb]/[0b]}
[2022-01-12T08:23:08,751][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430988] overhead, spent [5.3s] collecting in the last [5.8s]
[2022-01-12T08:23:11,727][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430989][73141] duration [2.7s], collections [1]/[3s], total [2.7s]/[23.8h], memory [5.1gb]->[4.9gb]/[7.7gb], all_pools {[young] [156mb]->[4mb]/[0b]}{[old] [4.9gb]->[4.9gb]/[7.7gb]}{[survivor] [39.7mb]->[51.5mb]/[0b]}
[2022-01-12T08:23:11,727][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430989] overhead, spent [2.7s] collecting in the last [3s]
[2022-01-12T08:23:17,146][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430992][73142] duration [2.6s], collections [1]/[3.4s], total [2.6s]/[23.8h], memory [5.2gb]->[4.9gb]/[7.7gb], all_pools {[young] [280mb]->[0b]/[0b]}{[old] [4.9gb]->[4.9gb]/[7.7gb]}{[survivor] [51.5mb]->[14mb]/[0b]}
[2022-01-12T08:23:17,146][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430992] overhead, spent [2.6s] collecting in the last [3.4s]
[2022-01-12T08:23:27,345][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430998][73144] duration [4.6s], collections [1]/[5.1s], total [4.6s]/[23.8h], memory [5.1gb]->[5gb]/[7.7gb], all_pools {[young] [208mb]->[0b]/[0b]}{[old] [4.9gb]->[4.9gb]/[7.7gb]}{[survivor] [34.7mb]->[52mb]/[0b]}
[2022-01-12T08:23:27,345][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430998] overhead, spent [4.6s] collecting in the last [5.1s]
[2022-01-12T08:23:37,276][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][430999][73145] duration [9.6s], collections [1]/[9.9s], total [9.6s]/[23.8h], memory [5gb]->[5gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [4.9gb]->[5gb]/[7.7gb]}{[survivor] [52mb]->[42mb]/[0b]}
[2022-01-12T08:23:37,276][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][430999] overhead, spent [9.6s] collecting in the last [9.9s]
[2022-01-12T08:23:39,436][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431000][73146] duration [1.9s], collections [1]/[2.1s], total [1.9s]/[23.8h], memory [5gb]->[5gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [5gb]->[5gb]/[7.7gb]}{[survivor] [42mb]->[24mb]/[0b]}
[2022-01-12T08:23:39,437][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431000] overhead, spent [1.9s] collecting in the last [2.1s]
[2022-01-12T08:23:44,837][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431001][73150] duration [4.6s], collections [4]/[5.4s], total [4.6s]/[23.8h], memory [5gb]->[5gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [5gb]->[5gb]/[7.7gb]}{[survivor] [24mb]->[26mb]/[0b]}
[2022-01-12T08:23:44,837][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431001] overhead, spent [4.6s] collecting in the last [5.4s]
[2022-01-12T08:23:50,155][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431004][73151] duration [2.3s], collections [1]/[3.3s], total [2.3s]/[23.8h], memory [5.3gb]->[5gb]/[7.7gb], all_pools {[young] [248mb]->[4mb]/[0b]}{[old] [5gb]->[5gb]/[7.7gb]}{[survivor] [26mb]->[52mb]/[0b]}
[2022-01-12T08:23:50,155][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431004] overhead, spent [2.3s] collecting in the last [3.3s]
[2022-01-12T08:24:02,966][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431005][73152] duration [11.9s], collections [1]/[12.8s], total [11.9s]/[23.8h], memory [5gb]->[5.1gb]/[7.7gb], all_pools {[young] [4mb]->[8mb]/[0b]}{[old] [5gb]->[5gb]/[7.7gb]}{[survivor] [52mb]->[20mb]/[0b]}
[2022-01-12T08:24:02,966][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431005] overhead, spent [11.9s] collecting in the last [12.8s]
[2022-01-12T08:24:10,346][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431006][73153] duration [6.7s], collections [1]/[7.3s], total [6.7s]/[23.8h], memory [5.1gb]->[5.1gb]/[7.7gb], all_pools {[young] [8mb]->[16mb]/[0b]}{[old] [5gb]->[5.1gb]/[7.7gb]}{[survivor] [20mb]->[52mb]/[0b]}
[2022-01-12T08:24:10,346][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431006] overhead, spent [6.7s] collecting in the last [7.3s]
[2022-01-12T08:24:25,734][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43796}] took [15180ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:24:25,734][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431007][73154] duration [14.9s], collections [1]/[15.3s], total [14.9s]/[23.8h], memory [5.1gb]->[5.2gb]/[7.7gb], all_pools {[young] [16mb]->[0b]/[0b]}{[old] [5.1gb]->[5.1gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:24:25,734][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431007] overhead, spent [14.9s] collecting in the last [15.3s]
[2022-01-12T08:24:25,740][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][HEAD][/_alias/win_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43814}] took [15180ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:24:41,600][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431008][73155] duration [15.5s], collections [1]/[15.8s], total [15.5s]/[23.8h], memory [5.2gb]->[5.2gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [5.1gb]->[5.2gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:24:41,600][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431008] overhead, spent [15.5s] collecting in the last [15.8s]
[2022-01-12T08:25:07,438][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431009][73157] duration [24.7s], collections [2]/[25.8s], total [24.7s]/[23.8h], memory [5.2gb]->[5.4gb]/[7.7gb], all_pools {[young] [0b]->[28mb]/[0b]}{[old] [5.2gb]->[5.3gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:25:07,438][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431009] overhead, spent [24.7s] collecting in the last [25.8s]
[2022-01-12T08:25:24,461][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431010][73158] duration [16.6s], collections [1]/[17s], total [16.6s]/[23.8h], memory [5.4gb]->[5.4gb]/[7.7gb], all_pools {[young] [28mb]->[20mb]/[0b]}{[old] [5.3gb]->[5.4gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:25:24,461][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431010] overhead, spent [16.6s] collecting in the last [17s]
[2022-01-12T08:25:29,480][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431011][73159] duration [4.5s], collections [1]/[5s], total [4.5s]/[23.8h], memory [5.4gb]->[5.4gb]/[7.7gb], all_pools {[young] [20mb]->[12mb]/[0b]}{[old] [5.4gb]->[5.4gb]/[7.7gb]}{[survivor] [52mb]->[30mb]/[0b]}
[2022-01-12T08:25:29,480][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431011] overhead, spent [4.5s] collecting in the last [5s]
[2022-01-12T08:25:39,566][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431012][73160] duration [9.5s], collections [1]/[10s], total [9.5s]/[23.8h], memory [5.4gb]->[5.5gb]/[7.7gb], all_pools {[young] [12mb]->[0b]/[0b]}{[old] [5.4gb]->[5.5gb]/[7.7gb]}{[survivor] [30mb]->[52mb]/[0b]}
[2022-01-12T08:25:39,566][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431012] overhead, spent [9.5s] collecting in the last [10s]
[2022-01-12T08:25:39,567][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43836}] took [9888ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:25:07,467][WARN ][o.e.c.s.MasterService ] [node-1] took [16s], which is over [10s], to compute cluster state update for [create-index-template [gl-failures-template], cause [api]]
[2022-01-12T08:25:56,426][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431013][73161] duration [16.2s], collections [1]/[16.7s], total [16.2s]/[23.8h], memory [5.5gb]->[5.6gb]/[7.7gb], all_pools {[young] [0b]->[192mb]/[0b]}{[old] [5.5gb]->[5.5gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:25:56,426][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431013] overhead, spent [16.2s] collecting in the last [16.7s]
[2022-01-12T08:26:12,530][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431014][73162] duration [15.7s], collections [1]/[16.2s], total [15.7s]/[23.8h], memory [5.6gb]->[5.7gb]/[7.7gb], all_pools {[young] [192mb]->[0b]/[0b]}{[old] [5.5gb]->[5.6gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:26:12,530][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431014] overhead, spent [15.7s] collecting in the last [16.2s]
[2022-01-12T08:26:24,859][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/lnx_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43814}] took [11928ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:26:24,859][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43828}] took [11928ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:26:24,933][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431015][73163] duration [11.8s], collections [1]/[12.4s], total [11.8s]/[23.8h], memory [5.7gb]->[5.9gb]/[7.7gb], all_pools {[young] [0b]->[144mb]/[0b]}{[old] [5.6gb]->[5.7gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:26:24,934][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431015] overhead, spent [11.8s] collecting in the last [12.4s]
[2022-01-12T08:26:34,515][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431016][73164] duration [9.2s], collections [1]/[9.5s], total [9.2s]/[23.8h], memory [5.9gb]->[5.8gb]/[7.7gb], all_pools {[young] [144mb]->[0b]/[0b]}{[old] [5.7gb]->[5.7gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:26:34,515][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431016] overhead, spent [9.2s] collecting in the last [9.5s]
[2022-01-12T08:26:34,519][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/lnx_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43792}] took [9255ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:26:42,599][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431017][73165] duration [7.6s], collections [1]/[8s], total [7.6s]/[23.8h], memory [5.8gb]->[5.8gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [5.7gb]->[5.8gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:26:42,599][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431017] overhead, spent [7.6s] collecting in the last [8s]
[2022-01-12T08:26:42,610][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][PUT][/gl-failures_0?master_timeout=30s&timeout=30s][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43834}] took [7874ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:26:42,627][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/_msearch?typed_keys=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43816}] took [7874ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:26:47,952][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431018][73166] duration [4.9s], collections [1]/[5.3s], total [4.9s]/[23.8h], memory [5.8gb]->[5.8gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [5.8gb]->[5.8gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:26:47,952][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431018] overhead, spent [4.9s] collecting in the last [5.3s]
[2022-01-12T08:27:12,827][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/alias/lnx_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43792}] took [24653ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:27:12,830][WARN ][o.e.c.s.MasterService ] [node-1] took [24.8s], which is over [10s], to compute cluster state update for [create-index-template [gl-failures-template], cause [api]]
[2022-01-12T08:27:12,905][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431019][73167] duration [24.3s], collections [1]/[24.8s], total [24.3s]/[23.9h], memory [5.8gb]->[6gb]/[7.7gb], all_pools {[young] [0b]->[48mb]/[0b]}{[old] [5.8gb]->[5.9gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:27:12,905][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431019] overhead, spent [24.3s] collecting in the last [24.8s]
[2022-01-12T08:27:22,698][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/gl-events/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43828}] took [9671ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:27:22,717][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431020][73168] duration [9.5s], collections [1]/[9.9s], total [9.5s]/[23.9h], memory [6gb]->[6gb]/[7.7gb], all_pools {[young] [48mb]->[4mb]/[0b]}{[old] [5.9gb]->[6gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:27:22,717][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431020] overhead, spent [9.5s] collecting in the last [9.9s]
[2022-01-12T08:27:41,687][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431021][73169] duration [18.6s], collections [1]/[18.9s], total [18.6s]/[23.9h], memory [6gb]->[6.1gb]/[7.7gb], all_pools {[young] [4mb]->[0b]/[0b]}{[old] [6gb]->[6.1gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:27:41,687][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431021] overhead, spent [18.6s] collecting in the last [18.9s]
[2022-01-12T08:27:41,697][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/win_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43828}] took [18994ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:27:41,724][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43792}] took [18794ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:27:49,367][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431022][73170] duration [5.5s], collections [1]/[7.6s], total [5.5s]/[23.9h], memory [6.1gb]->[6.1gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.1gb]->[6.1gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:27:49,367][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431022] overhead, spent [5.5s] collecting in the last [7.6s]
[2022-01-12T08:28:02,692][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431023][73171] duration [12.9s], collections [1]/[13.3s], total [12.9s]/[23.9h], memory [6.1gb]->[6.2gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.1gb]->[6.2gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:02,692][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431023] overhead, spent [12.9s] collecting in the last [13.3s]
[2022-01-12T08:28:08,292][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431024][73172] duration [5.3s], collections [1]/[5.6s], total [5.3s]/[23.9h], memory [6.2gb]->[6.3gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.2gb]->[6.2gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:08,293][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431024] overhead, spent [5.3s] collecting in the last [5.6s]
[2022-01-12T08:28:08,332][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/_msearch?typed_keys=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43860}] took [5409ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:28:24,672][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431025][73173] duration [15.9s], collections [1]/[16.3s], total [15.9s]/[23.9h], memory [6.3gb]->[6.4gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.2gb]->[6.3gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:24,672][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431025] overhead, spent [15.9s] collecting in the last [16.3s]
[2022-01-12T08:28:24,695][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][HEAD][/_alias/gl-events_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43846}] took [15969ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:28:33,801][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431026][73174] duration [8.7s], collections [1]/[9.1s], total [8.7s]/[23.9h], memory [6.4gb]->[6.4gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.3gb]->[6.4gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:33,801][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431026] overhead, spent [8.7s] collecting in the last [9.1s]
[2022-01-12T08:28:33,801][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43794}] took [8928ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:28:33,801][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/lnx_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43866}] took [8928ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:28:48,225][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431027][73175] duration [14.1s], collections [1]/[14.4s], total [14.1s]/[23.9h], memory [6.4gb]->[6.5gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.4gb]->[6.5gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:48,225][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431027] overhead, spent [14.1s] collecting in the last [14.4s]
[2022-01-12T08:28:53,480][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431028][73176] duration [4.9s], collections [1]/[5.2s], total [4.9s]/[23.9h], memory [6.5gb]->[6.6gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.5gb]->[6.5gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:53,480][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431028] overhead, spent [4.9s] collecting in the last [5.2s]
[2022-01-12T08:28:53,483][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/alias/win_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43856}] took [5054ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:28:56,890][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431029][73177] duration [3.1s], collections [1]/[3.4s], total [3.1s]/[23.9h], memory [6.6gb]->[6.7gb]/[7.7gb], all_pools {[young] [0b]->[4mb]/[0b]}{[old] [6.5gb]->[6.6gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:28:56,890][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431029] overhead, spent [3.1s] collecting in the last [3.4s]
[2022-01-12T08:29:02,018][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431030][73178] duration [2s], collections [1]/[5.1s], total [2s]/[23.9h], memory [6.7gb]->[6.7gb]/[7.7gb], all_pools {[young] [4mb]->[0b]/[0b]}{[old] [6.6gb]->[6.6gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:29:02,029][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431030] overhead, spent [2s] collecting in the last [5.1s]
[2022-01-12T08:29:03,676][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431031][73179] duration [1.4s], collections [1]/[1.6s], total [1.4s]/[23.9h], memory [6.7gb]->[6.7gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.6gb]->[6.7gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:29:03,676][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431031] overhead, spent [1.4s] collecting in the last [1.6s]
[2022-01-12T08:29:10,101][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431032][73180] duration [6s], collections [1]/[6.4s], total [6s]/[23.9h], memory [6.7gb]->[6.8gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.7gb]->[6.8gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:29:10,101][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431032] overhead, spent [6s] collecting in the last [6.4s]
[2022-01-12T08:29:10,114][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/win/_alias?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=false][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43856}] took [6215ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:29:10,129][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][POST][/_msearch?typed_keys=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43794}] took [6215ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:29:16,810][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431033][73181] duration [6.4s], collections [1]/[6.7s], total [6.4s]/[23.9h], memory [6.8gb]->[6.9gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.8gb]->[6.9gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:29:16,810][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431033] overhead, spent [6.4s] collecting in the last [6.7s]
[2022-01-12T08:29:16,831][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][HEAD][/_alias/gl-system-events_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43856}] took [6508ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:29:20,129][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431034][73182] duration [3s], collections [1]/[3.3s], total [3s]/[23.9h], memory [6.9gb]->[7gb]/[7.7gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [6.9gb]->[6.9gb]/[7.7gb]}{[survivor] [52mb]->[52mb]/[0b]}
[2022-01-12T08:29:20,129][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431034] overhead, spent [3s] collecting in the last [3.3s]
[2022-01-12T08:29:20,483][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [7893278480]
[2022-01-12T08:29:25,334][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [7893278480], after [7601774352], allocations [1], duration [4905]
[2022-01-12T08:29:25,343][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431036][73183] duration [3.5s], collections [1]/[4.2s], total [3.5s]/[23.9h], memory [7.3gb]->[7gb]/[7.7gb], all_pools {[young] [332mb]->[16mb]/[0b]}{[old] [6.9gb]->[7gb]/[7.7gb]}{[survivor] [52mb]->[30mb]/[0b]}
[2022-01-12T08:29:25,343][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431036] overhead, spent [3.5s] collecting in the last [4.2s]
[2022-01-12T08:29:28,101][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431037][73184] duration [2.4s], collections [1]/[2.7s], total [2.4s]/[23.9h], memory [7gb]->[7.1gb]/[7.7gb], all_pools {[young] [16mb]->[0b]/[0b]}{[old] [7gb]->[7gb]/[7.7gb]}{[survivor] [30mb]->[52mb]/[0b]}
[2022-01-12T08:29:28,101][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431037] overhead, spent [2.4s] collecting in the last [2.7s]
[2022-01-12T08:29:32,976][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] attempting to trigger G1GC due to high heap usage [7968775952]
[2022-01-12T08:29:52,523][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-1] GC did bring memory usage down, before [7968775952], after [7647302928], allocations [14], duration [19547]
[2022-01-12T08:29:52,524][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] handling request [null][GET][/_alias/graylog_deflector?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=true][Netty4HttpChannel{localAddress=/10.3.208.43:9200, remoteAddress=/10.3.208.82:43794}] took [16827ms] which is above the warn threshold of [5000ms]
[2022-01-12T08:29:52,525][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][young][431059][73185] duration [2.2s], collections [1]/[3.4s], total [2.2s]/[23.9h], memory [7.4gb]->[7.1gb]/[7.7gb], all_pools {[young] [328mb]->[0b]/[0b]}{[old] [7.1gb]->[7.1gb]/[7.7gb]}{[survivor] [52mb]->[12mb]/[0b]}
[2022-01-12T08:29:52,525][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][431059] overhead, spent [2.2s] collecting in the last [3.4s]

tmacgbay · January 12, 2022, 4:33pm

Ay-ya that is a lot of log to look through…

I would take a look at what the rotation is on all your indexes. You can increase the limit but there might be side effects. I am not sure what version of Graylog or Elastic you are on… or OS… or configuration… or… so I can’t post any more info… but you can now google on elastic shard limit …

monarch684 · January 12, 2022, 9:22pm

This took me down a rabbit hole, but indeed the shards were maxed. I was able to increase and also cleaned up a bunch of stuff in regards to this. I am now only using 654 shards.

system · January 26, 2022, 9:22pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog_deflector exists as an indexer and is not an alias Graylog Central (peer support)	4	5647	November 12, 2018
Elastic connection issues according to the server.log Graylog Central (peer support)	5	860	November 5, 2020
Unable to create index template graylog-internal Graylog Central (peer support)	3	6322	June 6, 2017
[Graylog_2.3.2] Shards and deflectors errors Graylog Central (peer support)	14	8143	January 1, 2018
Graylog first problem's Graylog Central (peer support)	3	784	September 24, 2018

Multiple Errors Between Graylog & Elasticsearch

Related topics