So under the hood, what is actually happening is Graylog ensures shards are between 20-50GB in size, as that’s the recommended shard size for OpenSearch. You are indeed correct that if you want to ensure that your logs are available in online storage, and searchable, then you set the minimum lifetime to 365 days, the maximum is how long before data will be forced rotated and the index options are implemented (delete index etc).
The maximum should only come into play in index sets which do not receive much data.
The Time Based Size Optimizing Rotation Strategy tries to rotate the index daily. It can however skip the rotation to achieve optimal sized indices by keeping the shard size between 20 and 50 GB. The optimization can delay the rotation within the range of the configured retention min/max lifetime. If an index is older than the range between min/max, it will be rotated regardless of its current size
I think this still needs to be cleaned up some, at least it’s confusing the rotation policy has settings borrowed from the retention policy. If you select do nothing for the retention policy i have no min/max sliders, but the rotation policy suggest i need to set sliders to configure the max lifetime of a index before it’s rotated.
I am confused on when it will actually starts skipping rotating the index daily to keep the shard size between 20-50GB. I assume it will always rotate the index between 20-5GB shard size no matter what. So if I’m getting a lot of data in, it will rotate the index multiple times a day to keep the shard size 20-50GB.
But if I’m getting very little data on day 1, say 5GB. what happens? Does it rotate daily, or does it wait multiple days until i get 20-50GB? From my understanding the index will stay open until it reaches the max lifetime or 20-50GB. If the index hits the max slider it will be rotated AND deleted at the same time. If you have delete index selected for the retention strategy.
I have no idea what the min slider does. Above it says to set the min slider to 365 to keep the logs for 1 year, but then says max slider is how long before the retention policy kicks in, and that’s what is actually removing the logs from searching… So that would suggest min/max should be the same value?
Your post sent me down a rabbit hole, but it proved to be very fruitful
I ended up reading any applicable github issues, internal confluence pages, forum posts, and official documentation, and then consulting with a dev here to fact check the things i either was confused about or got wrong. I put together this diagram that shows how the logic works:
This helped demystify a lot of things for me, such as “how often are these thresholds checked!”.
The rotation logic is checked once every 10 seconds. Meaning that once an index meets any of the criteria, it will be rotated.
The index is rotated for the following conditions, in this order:
Index Size (which is the size of all shards in that index) is greater than defined max index size (default 50GB)
Index Creation Date is older than “leeway” (Max Age Slider minus Min Age Slider, for example if Max Age is 10, and Min age is 5, the leeway is 5 days)
Index Create Date is at least 1 day old, Index Size is at least large than defined min index size (default 20GB)
To answer a couple of your questions specifically:
If you select do nothing for the retention policy i have no min/max sliders, but the rotation policy suggest i need to set sliders to configure the max lifetime of a index before it’s rotated
Index is rotated when index create day is at least as old as the rotation period (1day) AND index size is larger than minimum index size (20GB by default)
if I’m getting very little data on day 1, say 5GB. what happens
the index won’t rotate, see diagram.
Let me know if any of this doesn’t make sense or if you have any other questions.
I’m still hung up on this, pretty sure this is not right still or might need to be re-worked.
Index Creation Date is older than “leeway” (Max Age Slider minus Min Age Slider, for example if Max Age is 10, and Min age is 5, the leeway is 5 days)
Index Create date > (Max Age - Min Age)…
What issue is this trying to solve?
If i set my min to 355 and max to 365… That is 10 days. Why are we rotating the index 10 days after we create it?
In your diagram you have a check in the bottom right to verify the index size is at least 20gb before we rotate, so we don’t rotate an index that is too small, but this step can totally bypass that.
If I have 10 days leeway, i could get 1 log message in that entire time, and according to this i would rotate the index with 1 log message.
