We are running graylog 4.0.2+1987d10 with Java 1.8.0_275 on RHEL 8.3 with Kernel 4.18.0-240.1.1.el8_3.x86_64 on a physical Server.
Elasticsearch version 7.11.0 runs on 3 physical Servers. Graylog was upgraded from 3.3.9+abab7dc to 4.0.1+6a0cc0b in January and in February to 4.0.2+1987d10. ES was upgraded from 6.8.13 to 7.10.2 in February and to 7.11.0 today.
Default Index set is configured to store 20 Mill entries per index, rotate strategy is by message count and use 12 indices.
Since graylog 4, the automatic index rotation does not work. Manual rotate is working, which was done last week:
2021-02-08T16:09:02.488+01:00 INFO [MongoIndexSet] Cycling from <graylog2_146> to <graylog2_147>.
2021-02-08T16:09:02.488+01:00 INFO [MongoIndexSet] Creating target index <graylog2_147>.
2021-02-08T16:09:02.758+01:00 INFO [MongoIndexSet] Waiting for allocation of index <graylog2_147>.
2021-02-08T16:09:03.406+01:00 INFO [MongoIndexSet] Index <graylog2_147> has been successfully allocated.
2021-02-08T16:09:03.406+01:00 INFO [MongoIndexSet] Pointing index alias <graylog2_deflector> to new index <graylog2_147>.
2021-02-08T16:09:03.462+01:00 INFO [SystemJobManager] Submitted SystemJob <949a6950-6a1f-11eb-a473-48df37a67540> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2021-02-08T16:09:03.462+01:00 INFO [MongoIndexSet] Successfully pointed index alias <graylog2_deflector> to index <graylog2_147>.
2021-02-08T16:09:23.978+01:00 INFO [SetIndexReadOnlyJob] Flushing old index <graylog2_146>.
2021-02-08T16:09:33.593+01:00 INFO [SetIndexReadOnlyJob] Setting old index <graylog2_146> to read-only.
2021-02-08T16:09:33.659+01:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_146.
2021-02-08T16:09:33.666+01:00 INFO [OptimizeIndexJob] Optimizing index <graylog2_146>.
2021-02-08T16:09:36.889+01:00 INFO [MongoIndexRangeService] Calculated range of [graylog2_146] in [3227ms].
2021-02-08T16:09:36.891+01:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_146.
2021-02-08T16:09:36.896+01:00 INFO [SystemJobManager] SystemJob <949a6950-6a1f-11eb-a473-48df37a67540> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 3434ms.
2021-02-08T16:12:54.980+01:00 INFO [AbstractIndexCountBasedRetentionStrategy] Number of indices (13) higher than limit (12). Running retention for 1 indices.
2021-02-08T16:12:55.006+01:00 INFO [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog2_135>
2021-02-08T16:12:56.267+01:00 INFO [DeletionRetentionStrategy] Finished index retention strategy [delete] for index <graylog2_135> in 1260ms.
But the Indices does not automatically get rotated. What could causing this? Or how can this be debuged?
checking the Graylog log when the log should get rotated might help. I guess that some API response has changed with ES 7.11 that was used to see the document count or similar and that might be the reason.
Checking all logs and looking for errors might help here.
Hey @jan
We had this problem with ES 7.10 too.
The interesting thing is, on the default index site, the no of messages in the active write index is listed: ## graylog2_147 active write index Contains messages up to a few seconds ago (7.9GiB / 21,176,170 messages)
There are not any errors concerning index rotation in the logs.
Can a “check for rotation” be triggered manually?
Is there a special Debug level/option for rotation?
Hello @jan
Unfortunately this does not help
When I trigger a “re-calculate index range” from the Default Index set (Maintenance → Recalculate index ranges) all closed/old indices get recalulatet, but not the active one … Via the API and Rebuild/sync index range information (http://127.0.0.1:9000/api/system/indices/ranges/graylog2_147/rebuild) the active index got rebuild. But that does not change anything … ## graylog2_147 active write index Contains messages up to a few seconds ago (9.6GiB / 26,512,257 messages)
2021-02-18T08:21:23.991+01:00 INFO [SystemJobManager] Submitted SystemJob <e7fc9270-7123-11eb-82da-48df37a67540> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2021-02-18T08:21:24.013+01:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_147.
2021-02-18T08:21:26.592+01:00 INFO [MongoIndexRangeService] Calculated range of [graylog2_147] in [2576ms].
2021-02-18T08:21:26.593+01:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_147.
2021-02-18T08:21:26.593+01:00 INFO [SystemJobManager] SystemJob <e7fc9270-7123-11eb-82da-48df37a67540> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 2602ms.
I do see a lot of these warnings in the logs: 2021-02-18T11:20:23.605+01:00 WARN [RestClient] request [POST https://u123.local:9200/_msearch?typed_keys=true] returned 1 warnings: [299 Elasticsearch-7.11.0-8ced7813d6f16d2ef30792e2fcde3e123795ee04 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Use new java.time date format specifiers."]
These are from the old indices, createt with ES 6.8 and Graylog 3.x, right? They should go away, when the old indices got deletet. This should not interfere the current index rotation. Is this correct?
