Graylog 4.0 Elasticsearch Index does not roatate by message count

[miek (Mike Murphy)]

Hello,
I’ve been trying to figure out why my default index does not rotate. Actually, any index set that is set to rotate by document count is not working. I have 25 Indexes with 4 shards set to rotate every 20,000,000 documents / messages, but it just doesn’t. I have been able to rotate these manually with no issues. Prior to the manual rotation it showed 1 index with about 1.2 billion messages.

I have indexes that are set to rotate by time and they appear to be working as expected. Any ideas? Is this a known bug? I assume a document and a message are two terms for the same thing?

This is a bare metal install with Elastisearch, mongoDB, and Graylog all on the same box.

Thanks

[miek (Mike Murphy)]

So, it looks like this is a bug in Graylog - Elastisearch. The workaround is simply not to create indexes that rotate on document count. Setting up indexes that rotate on index size, or that rotate by time seam to work just fine.

[dickinsonzach]

Good morning, do you mean this setting:

From what you’ve experience when my document count hits 20,000,000 it will not roll?

Thank you, Zach.

[roxor (John Soros)]

That is correct. I’m not sure what combination of software are affected, but with elasticsearch 7.10.2 and graylog 4.0.1, document-count based index rotation does not work.

[miek (Mike Murphy)]

Hi,
Sorry for the late response. Yes, that is what I mean. It appears to not work with document count. If you switch the type of rotation to index size or time period, it functions as expected.

[miek (Mike Murphy)]

Thanks for confirming that John,