We’re running Graylog 5.0.8 (out of date, I know) on RHEL 9.2. Last week, we updated the index set for our Windows event logs, changing the max number of indices from 44 to 50, with no other changes. The max size per index is set to 4 GiB, so we would expect to see a total max size of 200 GiB for this index set.
However, since that change was made, it has not rotated the current write index, which has now grown to be over 260 GiB in size. We have tried Recalculate index ranges and Rotate active write index but they seem to have no effect. We have restarted Graylog and even rebooted the server itself, but it just goes back to the same write index again. We really want to get this index to rotate per its criteria, so we don’t run out of disk space.
A screenshot is below for reference. Any ideas are greatly appreciated!
Thanks, Joel! This may have poin ted us in the right direction. I found this in server.log:
2025-09-10T11:12:03.001-04:00 INFO [Indices] Successfully ensured index template windows_event-template
2025-09-10T11:12:03.002-04:00 WARN [Indices] Couldn't create index windows_event_3535. Error: Unable to create index windows_event_3535
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: Unable to create index windows_event_3535
at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:155) ~[?:?]
at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:112) ~[?:?]
at org.graylog.storage.opensearch2.IndicesAdapterOS2.create(IndicesAdapterOS2.java:157) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:211) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:79) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:127) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:91) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:87) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:94) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog.shaded.opensearch2.org.opensearch.OpenSearchStatusException: OpenSearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [997]/[1000] maximum shards open;]
at org.graylog.shaded.opensearch2.org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2081) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2058) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1747) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1711) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.IndicesClient.create(IndicesClient.java:159) ~[?:?]
at org.graylog.storage.opensearch2.IndicesAdapterOS2.lambda$create$3(IndicesAdapterOS2.java:157) ~[?:?]
at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:110) ~[?:?]
... 15 more
Suppressed: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [PUT], host [http://127.0.0.1:9200], URI [/windows_event_3535?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [997]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [4] total shards, but this cluster currently has [997]/[1000] maximum shards open;"},"status":400}
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:375) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:345) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:320) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1747) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1711) ~[?:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.IndicesClient.create(IndicesClient.java:159) ~[?:?]
at org.graylog.storage.opensearch2.IndicesAdapterOS2.lambda$create$3(IndicesAdapterOS2.java:157) ~[?:?]
at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:110) ~[?:?]
at org.graylog.storage.opensearch2.IndicesAdapterOS2.create(IndicesAdapterOS2.java:157) ~[?:?]
at org.graylog2.indexer.indices.Indices.create(Indices.java:211) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:79) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:127) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:91) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:87) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:94) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-09-10T11:12:03.003-04:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <windows_event_3535>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:293) ~[graylog.jar:?]
at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:79) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:127) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:91) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:87) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:94) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
It looks like we need to free up a few shards in order for this rotation to work. Or is there a way to increase the max shards without a restart?
We were able to delete an unused index set, which freed up 4 shards. Once that was done, the Windows event log index rotated on its own.
We subsequently discovered that none of our index sets had rotated in the last 13 days, due to this same issue. Each one was set to use 4 shards per index, but we were 3 away from the limit. We’re running a standalone node, so we changed each index one at a time, reducing the shards from 4 to 2 and then forcing a rotation. This had the effect of reducing the total number of shards in use, as each set had at least one index which aged out upon rotation, and freed up 4 more shards.
We now seem to be in much better shape, and I think our disk usage should equalize and eventually fall as these indices which were stuck for 2 weeks age out and get deleted.
With a single node there aren’t really any advantages of having more than 1 shard per index (they are helpful when you can load balance across multiple)
The ideal shard/index(because of above) is around 20GB-50GB, at sizes less than that, you are adding overhead.
If you upgrade to a newer version of Graylog 6+ then there is an option where you just tell Graylog how long you want to keep the data for, and it will do everything else for you to optimize it.
This is very helpful to know. We’re out of the woods as far as the upper limit is concerned, but we can further reduce out number by going from 2 to 1 on each index set.
This is something we’ll need to look into some more. Most of our sets will have far less than 20GB per shard at each rotation, simply due to the nature of the logs and the retention policies in place. What is the nature of the overhead incurred by having shards with less than 20GB (or more than 50GB)? Are they pre-allocated blocks, thus wasting space in each one if you don’t use most of it?
That sounds fantastic. We’re really hoping to get an upgrade to Graylog 6 underway in the not-so-distant future, when we have the time to work on it. It sounds like version 6 will solve a lot of the issues we run into with 5.
Its not wasting storage space, it’s more that a shard uses an amount of ram and cpu to keep track of it and so you want as few as possible, but then for other reasons don’t want them too big.
Definitely much easier when you don’t have to think about it
