We are creating new indexes every 12 hours.
We backup indexes daily using snapshots.
Deletion of indexes occur after 90 days.
What we are finding is that new data is ending up in old indexes.
If we delete the old indexes then we will be losing new data.
The bulk of the messages are in the period where the index was the active write index however it is very concerning that we will be losing some messages when we delete old indexes.
Is this normal behavior?
How do we ensure that old indexes only contain messages from when the index was the active write index?
Also it would greatly help if graylog allowed the use of the index creation date in UTC for the index number so instead of graylog_123 it would be graylog_2018091809T212418