Index rotation stopped working

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
All my index sets stopped working 22 days ago. They are supposed to rotate daily.

2. Describe your environment:

  • OS Information: Debian 12

  • Package Version: Graylog 6.0.4 // MongoDB 7.0.12

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
Tried to restart the mongod service, who failed. Add to remove the /tmp/mongodb-27017.sock file to start it again.

When I try to manually rotate the index via the GUI, I can see Cycling deflector for index set <66acea568c0cef5527621333>. Reason: REST request, but I get the following logs from graylog-server/server.log :

2024-10-04T10:24:44.883+02:00 ERROR [IndexRotationThread] Couldn’t point deflector to a new index
java.lang.RuntimeException: Could not create new target index <windows_43>.
at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:300) ~[graylog.jar:?]
at org.graylog2.indexer.rotation.common.IndexRotator.rotate(IndexRotator.java:74) ~[graylog.jar:?]
at org.graylog2.datatiering.rotation.DataTierRotation.rotate(DataTierRotation.java:70) ~[graylog.jar:?]
at org.graylog2.datatiering.hotonly.HotOnlyDataTieringOrchestrator.rotate(HotOnlyDataTieringOrchestrator.java:66) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:116) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:95) ~[graylog.jar:?]
at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]

Thanks for your help :slight_smile:

2

Do you see any other error message prior to this one?
Index creation failed - we need to figure out why. There should be an error message like this:
"Couldn't create index {}. Error: {}"

3

Hi, thanks for replying.

I just noticed that my disk watermark [95%] exceeded when the problem happened. The day I got the alert, I just extended the disk, but the all my active indexes seem to be stuck now, and won’t rotate…

4

I suspect your ES/OS is now in read-only mode:

5

Yes… I tried that. The following command :

PUT */_settings?expand_wildcards=all
{
“index.blocks.read_only_allow_delete”: null
}

It got correctly aknowledged, but it doesn’t change anything sadly. :confused:

6

OK, all my shards were used. Increased it with

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'

And every index rotated. Thanks for your help!

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.