ES not consuming enough disk space at maximum retention

[H2Cyber]

I have a two VM setup :

• First VM with GL and MongoDB
• Second VM with ES. This one has a seperate data partition of 950Gb which is dedicated for storing events

I have read that 30/35% of free disk space should always be left empty for ES. So this leaves me with 65% of the total 950Gb disk space, which is ~600Gb.
I have opted for the follwoing Index setup :

• Index shards 1
• Index replicas : 0
• Rotation strategy : Index size
• Retention strategy : Delete Index
• Max size per index : 12 Gib
• Max number of indices : 44 (I have estimated this value by deviding 600Gb by the max size per index which is 12)

Given the setup above, I would assume that at maximum retention ES would be using 44 indices 12Gb each, ie 528 Gb.
However, and as can be seen in the screenshot below, when at maximum retention (all 44 indices used), I am only using 308Gb of space, leaving almost 70% of free space (which is a waste of useful storage).

Could someone kindly advice is something is wrong with my reasoning ? Why is my ES setup only consuming 308Gb of space at maximum retention ?

[jan (Jan Doberstein)]

he @H2Cyber

it looks like your data can be compressed quite well … the calculation is done on indices with the data not getting what is called a force_merge. This is like the good old defragementation on those old FAT disks… where it did matter where the data is stored on the disk.

In addition the force_merge will compress. Depending on the data this can be save you lots of space on disk. As you can see the rotation is done when the 12GB is reached - in your Graylog logs you should see something like “running optimization” or similar and after a while the return of the command.