No index rotation after upgrade Graylog to v5


I’ve got index creation, rotation and deletion issue after upgrading my Graylog, Mongodb and Elasticsearch to work on v5.

1. Describe your incident:
Graylog index rotation didn’t work after Graylog upgrade.
Graylog just puts everything in the last indice before upgrade.

Index has reach it’s limits and I have no logs anymore : Number of documents in the index can't exceed ....
When I try a rotation, I’ve got simply these types of logs from Elasticsearch : [...] adding template [linux_year_template] for index patterns [linux_year_*] [...].

2. Describe your environment:
OS Information:
This is a cluster with 3 Elasticsearches nodes and 3 Graylog nodes.

Elasticsearch and Mongod are on Ubuntu 20.04.2 LTS physical servers
Graylog are on Debian 11 VM

All servers are in the same subnet.

  • Package Version:
    Graylog : 5.0.7-1
    Mongod : 6.0.6
    Elasticsearch : 7.10.2

  • Service logs, configurations, and environment variables:
    Graylog :
    index [linux_year_48], type [_doc], id [], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index can't exceed [2147483519]

Elasticsearch when I try to rotate index :

{"type": "server", "timestamp": "", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "": "", "": "", "message": "adding template [linux_year-template] for index patterns [linux_year_*]", "cluster.uuid": "", "": ""  }

3. What steps have you already taken to try and solve the problem?
I’ve tried to rotate index on Graylog
I’ve tried to create a new index set but it’s not working
I’ve tried to rotate index through Elasticseatch CLI but I didn’t find out how to do

4. How can the community help?
I’m searching to fix this issue, I need index rotation, creation and deletion to work.
Can you help me for that case please ?

Best regards.

Can you share your index template settings and rotation settings for linux_year_ index set? Via System/Indices


Also can you confirm if this message is correct or not?

Number of documents in the index can’t exceed [2147483519]

Do you have that many messages in a single index?

Also are you using multiple index sets and do all of them have this same issue, or are you only using a single index set?


Thank you for your answer
I’m sorry but as new user on this forum I cannot send any pictures.

Please find below index set configuration :

  • 15 days rotation
  • deletion after 26 index rotations
  • 1 shard
  • 1 replica

I confirm that the message is this one : Number of documents in the index can’t exceed [2147483519].

I also confirm that I also have a lot of messages in the last index because rotation is not working anymore. I’m speaking about 1,5To messages for this last index made of 2147483519 messages.

I am using multiple index sets (27) and all have the same issues since Graylog have been updated to version 5 (I didn’t check each but some of them). The others are not full of messages, but there is no more index rotation.

I’ve made a bash script that allow me to output indexes name, age in days and index timestamp from Elasticsearch. There, I can see that before May 17th rotation was working ; one rotation each 15 days as configured. And then, after the upgrade, no more rotation : current index is always the 48th which have May the 17th as timestamp.

When I try to force rotation through Graylog “Rotate active write index” button (system > indices > my_index and top right button Maintenance) nothing happens, except the same message on Elasicsearch logs :

{"type": "server", "timestamp": "", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "": "", "": "", "message": "adding template [linux_year-template] for index patterns [linux_year_*]", "cluster.uuid": "", "": ""  }

When I try to create a new index set it’s not working.

It’s like if Graylog has no control on Elasticsearch index management anymore.
I’m thinking about rollback to previous version, but I don’t want to stay in older versions.

Thank you for your help.

Best regards.

Are you able to share the last 100-200 lines of your graylog server.log as well as the elasticsearch server log?

I’m very curious!

If they are too long you can use something like .


I’ve searched further when exporting logs for you and I found the issue with that log message : this action would add [2] total shards, but this cluster currently has [3056]/[3000] maximum shards open;.
This log message was drowned in these very numerous other messages : Number of documents in the index can’t exceed [2147483519].

I’ve increesed the number of allowed shards : curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'
I also deleted 365 days old shards.

Thank you for your help, this case is solved !!

Best regards.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.