Index set not filling up completely before rotating

I am seeing this weird behavior where the index set rotates before it reaches the size that is set. For e.g. I had the index set size set to 1 gb before rotating to the next index set but it started getting rotated at around 400mb. I tried changing the size of index set to 530mb but it still rotates short of that. I will attach screenshots below.


This is the index set configuration

I have checked the elasticseach cluster and status is green.
In case any of you has seen this before any help will be much appreciated!
2. Describe your environment:

  • OS Information: Red Hat Enterprise Linux Server release 7.9 (Maipo)

  • Package Version: 4.3.15+17ed3ac, codename Noir

  • Service logs, configurations, and environment variables:n/a

Thanks

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

I’m curious what indexer are you using? (Elasticsearch/OpenSearch and version)

We no longer recommend using size based rotation, but i suspect this could possibly be due to index compression. More details about your environment are needed though.

It also looks like you are running an old version of Graylog from a few years ago. We’ve just released Graylog 6.1. If you get a chance you should check it out :slight_smile:

Hi Grav

You have Index optimization enabled on this index set. This is a compression process that runs after an index has been rotated, which in my experience typically compresses an index to around 60% of its original size (but note compression rate varies depending on the data within the index).

If your index is rotating at 1GB, it seems within ballpark that the compressed data is ~400MB.

If you disable index optimization, you should find your rotated indices are closer to the 1GB size.

image

thank you for your responses, I have set the index to rotate based on P1D and that is working very well.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.