Graylog 6.0 and index retention / deletion

rm8d · September 14, 2024, 7:59am

Describe your incident:

I have Graylog 6.0.6 running with default configuration. I have configured the ‘Data Tiering’ as follows:

Max. days in storage → 10
Min. days in storage → 7

I have an index which is capturing logs from pfsense, and over time, the active write index is rolled over as expected, however, the index deletion process appears to not be working.

What needs to be configured / enabled to ensure that the oldest index based on the retention settings is removed? I would have thought that after 10 days, the index would be removed.

The screenshot below shows the oldest index.

Describe your environment:
Ubuntu 22.04
Graylog 6.0.6

What steps have you already taken to try and solve the problem?

I’ve recreated the index from scratch, but the index rotation still does not run through a deletion process.

Wine_Merchant · September 16, 2024, 10:02am

Hello @rm8d

Could you post a picture of the index set config and do the server logs point to an issue at the point of rollover?

rm8d · September 25, 2024, 12:46pm

Hi @Wine_Merchant, pls see images below.

Do let me know which ‘server logs’ you are referring to?

rm8d · September 25, 2024, 12:52pm

@Wine_Merchant, see the ‘server.log’ below

        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:94) ~[graylog.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-25T22:51:02.200+10:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2024-09-25T22:51:02.203+10:00 INFO  [MongoIndexSet] There is no index target to point to. Creating one now.
2024-09-25T22:51:02.205+10:00 INFO  [MongoIndexSet] Cycling from <none> to <gl-failures_0>.
2024-09-25T22:51:02.205+10:00 INFO  [MongoIndexSet] Creating target index <gl-failures_0>.
2024-09-25T22:51:02.206+10:00 WARN  [Indices] Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
java.lang.IllegalStateException: No index template provider found for type 'failures'
        at org.graylog2.indexer.IndexMappingFactory.resolveIndexMappingTemplateProvider(IndexMappingFactory.java:60) ~[graylog.jar:?]
        at org.graylog2.indexer.IndexMappingFactory.createIndexMapping(IndexMappingFactory.java:52) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.buildTemplate(Indices.java:214) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.ensureIndexTemplate(Indices.java:191) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.create(Indices.java:235) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:299) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:267) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:160) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:94) ~[graylog.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-09-25T22:51:02.211+10:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:300) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:267) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:160) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:94) ~[graylog.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]

rm8d · September 26, 2024, 1:04pm

@Wine_Merchant I’ve tried the following in an attempt to remove the read only option on the index, post a low watermark issue.

curl -X PUT ‘http://localhost:9200/_all/_settings?pretty’
-H ‘Content-Type: application/json’
-d ‘{“index.blocks.read_only_allow_delete”: null}’

I’ve had to delete the indexes manually.

Any ideas what else i can check to help the rotation?

Wine_Merchant · September 30, 2024, 10:56am

Hey @rm8d

I’ve been away thus not replying. The logs you posted are not related to the index set you are having issues with, the error you posted is related to the enterprise version of GL. If you wanted logs concerning this index set, try manually rotating the index and looking at the logs generated during that job.

What happens is you switch back to legacy rotation method and select the delete?

rm8d · October 1, 2024, 11:35am

@Wine_Merchant all good.

Rotating the log manually as show below.

Resulted in the following in the server.log.

2024-10-01T21:04:02.519+10:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:300) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:267) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:160) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:94) ~[graylog.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-10-01T21:04:05.442+10:00 INFO  [DeflectorResource] Cycling deflector for index set <66ca518986eb6423b7abfdce>. Reason: REST request.
2024-10-01T21:04:05.448+10:00 INFO  [MongoIndexSet] Cycling from <pfsense-fw02-index__12> to <pfsense-fw02-index__13>.
2024-10-01T21:04:05.448+10:00 INFO  [MongoIndexSet] Creating target index <pfsense-fw02-index__13>.
2024-10-01T21:04:05.798+10:00 INFO  [Indices] Successfully ensured index template pfsense-fw02-index_-template
2024-10-01T21:04:06.864+10:00 INFO  [MongoIndexSet] Waiting for allocation of index <pfsense-fw02-index__13>.
2024-10-01T21:04:06.898+10:00 INFO  [MongoIndexSet] Index <pfsense-fw02-index__13> has been successfully allocated.
2024-10-01T21:04:06.898+10:00 INFO  [MongoIndexSet] Pointing index alias <pfsense-fw02-index__deflector> to new index <pfsense-fw02-index__13>.
2024-10-01T21:04:07.157+10:00 INFO  [SystemJobManager] Submitted SystemJob <e0bd5430-7fe4-11ef-a260-00155d015029> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2024-10-01T21:04:07.158+10:00 INFO  [MongoIndexSet] Successfully pointed index alias <pfsense-fw02-index__deflector> to index <pfsense-fw02-index__13>.
2024-10-01T21:04:12.514+10:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2024-10-01T21:04:12.516+10:00 INFO  [MongoIndexSet] There is no index target to point to. Creating one now.
2024-10-01T21:04:12.517+10:00 INFO  [MongoIndexSet] Cycling from <none> to <gl-failures_0>.
2024-10-01T21:04:12.517+10:00 INFO  [MongoIndexSet] Creating target index <gl-failures_0>.
2024-10-01T21:04:12.517+10:00 WARN  [Indices] Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:300) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:267) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:160) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:94) ~[graylog.jar:?]
        at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:91) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-10-01T21:04:37.180+10:00 INFO  [SetIndexReadOnlyAndCalculateRangeJob] Flushing old index <pfsense-fw02-index__12>.
2024-10-01T21:04:38.397+10:00 INFO  [SetIndexReadOnlyAndCalculateRangeJob] Setting old index <pfsense-fw02-index__12> to read-only.
2024-10-01T21:04:38.697+10:00 INFO  [SystemJobManager] Submitted SystemJob <f38a4190-7fe4-11ef-a260-00155d015029> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2024-10-01T21:04:38.705+10:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index pfsense-fw02-index__12.
2024-10-01T21:04:38.706+10:00 INFO  [OptimizeIndexJob] Optimizing index <pfsense-fw02-index__12>.
2024-10-01T21:04:40.489+10:00 INFO  [MongoIndexRangeService] Calculated range of [pfsense-fw02-index__12] in [1783ms].
2024-10-01T21:04:40.492+10:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index pfsense-fw02-index__12.
2024-10-01T21:04:40.496+10:00 INFO  [SystemJobManager] SystemJob <e0bd5430-7fe4-11ef-a260-00155d015029> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 3335ms.
2024-10-01T21:04:42.517+10:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2024-10-01T21:04:42.518+10:00 INFO  [MongoIndexSet] There is no index target to point to. Creating one now.
2024-10-01T21:04:42.519+10:00 INFO  [MongoIndexSet] Cycling from <none> to <gl-failures_0>.
2024-10-01T21:04:42.519+10:00 INFO  [MongoIndexSet] Creating target index <gl-failures_0>.
2024-10-01T21:04:42.519+10:00 WARN  [Indices] Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
java.lang.IllegalStateException: No index template provider found for type 'failures'

rm8d · October 1, 2024, 11:42am

@Wine_Merchant Thanks for your help so far. See below legacy ’ Rotation and Retention’.

I’ll keep an eye on it and see if the indexes are rotated.

rm8d · October 8, 2024, 12:06pm

@Wine_Merchant unfortunately the above changes have not worked. Any other ways to debug this issue?

Wine_Merchant · October 8, 2024, 1:14pm

If you search your log file (/var/log/graylog-server/server.log) for the term “Delete” do you see entries referencing deletion jobs post rotation of an index?

system · October 22, 2024, 1:15pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How does Graylog delete old indexes when time based rotation is configure Graylog Central (peer support)	3	4375	July 26, 2019
Changin Index rotation type Graylog Central (peer support)	4	1572	December 13, 2017
Retaining logs for 6 months Graylog Central (peer support)	3	2698	March 9, 2020
Graylog Retention Strategy Not working Graylog Central (peer support)	3	2070	October 19, 2017
Indices, rotation, disk space Graylog Central (peer support)	10	8662	March 21, 2019

Graylog 6.0 and index retention / deletion

Related topics