Help in understanding the new "Index Time Size Optimizing" strategy

[Kaeltis]

Hi,

I’ve just upgraded to Graylog 5.1 and migrated from Elasticsearch to Opensearch 2.
It seems the new “Index Time Size Optimizing” strategy is the new default for rotating indices, however I’m not quite sure on how it works and how to configure it for my needs in terms of retaining older data.
I know I can still use the other strategies, but avoiding issues with indices being too small or too large seems like a good reason to move to the new strategy.

With the old strategies, I can for example select the following:

Index Strategy: Index Time
Rotation Period: P1M
Retention Strategy: Delete Index
Max number of indices: 12

This guarantees I can access logs from around 12 months ago, the same with the other strategies with some fiddling or an increase of “Max number of indices”.

The new “Index Time Size Optimizing” does not have an option for “Max number of indices”, so how would I configure this to keep the logs for at least a year?
Is it as simple as setting the the “Lifetime in days” minimum slider to 365?
The maximum slider would be 730 if I wanted to keep the logs for a maximum of two years?

[kpearson (Kyle Pearson)]

So under the hood, what is actually happening is Graylog ensures shards are between 20-50GB in size, as that’s the recommended shard size for OpenSearch. You are indeed correct that if you want to ensure that your logs are available in online storage, and searchable, then you set the minimum lifetime to 365 days, the maximum is how long before data will be forced rotated and the index options are implemented (delete index etc).

The maximum should only come into play in index sets which do not receive much data.

[Kaeltis]

Thanks for explaining!