Indices only using 2/3 of configured index size

[ErikHolgersson (Erik W.)]

Hello all,

as the title states, I have the problem that my Graylog indices only use about two thirds of their configured index size before rotation.

Index-Set settings:

Index-Set in reality:

Our Graylog runs on Kubernetes using the official image with an ElasticSearch-HTTP-Node and 4 ElasticSearch-Datanodes. I did not change any settings inside ElasticSearch apart from the necessary ones to create a cluster and assign roles.

I am aware that ElasticSearch compresses indices and that the active write index will be a bit bigger than a comparable closed index, both of which mean that the index will be below the assigned maximum size after rotation. However, not even using 11GB when the configured index size is 15GB seems oddly inefficient to me.

[jan (Jan Doberstein)]

he @ErikHolgersson

the size is measured by Graylog and the open index reports to be at the configured size Graylog will rotate and run a force_merge on that index. This can compress the indices in that way if the data allows that.

Cause Graylog can’t know how much data can be compressed the data and the elasticsearch version can have this kind of result.