My graylog elasticsearch indices have been deleted for no reason

[nosmoking1210 (luannguyen2022)]

Hi guys,
I have a cluster graylog with 3 nodes
Node1: graylog master, elastic, mongodb
Node2: graylog, elastic, mongodb
Node3: graylog, elastic, mongodb
All of them are clusterd.
Recently, I got a problem, when my indices in elasticsearch have been delete for no reason (I mean I dont know why).
I have check index retention and rotation policy, they are fine
Index retention strategy: Delete
Index rotation strategy: Document count (20M docs)
Max number of indices: 20

I’m pretty sure that I have enough storage for that (200GB each node), and the deletion always happen when I have graylog_0, graylog_1, graylog_2, graylog_3, graylog_4

This is what I found when all indices have been gone

server.log:2022-07-11T12:37:46.627Z INFO  [IndexRangesCleanupPeriodical] Removing index range information for unavailable indices: [gl-failures_5, gl-failures_4, gl-failures_3, gl-failures_2, gl-failures_1, graylog_1, graylog_4, graylog_3, gl-failures_6, graylog_2]

I cant find any reason which my elasticsearch indices have been delele, Can you guys give me some clues.

[gsmith]

Hello && welcome @nosmoking1210

Yes this is some Dark Graylog Magic.

@nosmoking1210

INFO [IndexRangesCleanupPeriodical]

Graylog goes through a clean check for index ranges, How do you know those indices have been deleted? Are they removed on the Web UI?

I’m assuming any configuration you made for these indices are made from the Web UI, correct?

Perhaps check Elasticsearch.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'  

Check template settings

curl -X GET "localhost:9200/graylog_6?pretty"