No index rotation after upgrade Graylog to v5

[VirtualFloppy]

Hello,

I’ve got index creation, rotation and deletion issue after upgrading my Graylog, Mongodb and Elasticsearch to work on v5.

1. Describe your incident:
Graylog index rotation didn’t work after Graylog upgrade.
Graylog just puts everything in the last indice before upgrade.

Index has reach it’s limits and I have no logs anymore : Number of documents in the index can't exceed ....
When I try a rotation, I’ve got simply these types of logs from Elasticsearch : [...] adding template [linux_year_template] for index patterns [linux_year_*] [...].

2. Describe your environment:
OS Information:
This is a cluster with 3 Elasticsearches nodes and 3 Graylog nodes.

Elasticsearch and Mongod are on Ubuntu 20.04.2 LTS physical servers
Graylog are on Debian 11 VM

All servers are in the same subnet.

• Package Version:
  Graylog : 5.0.7-1
  Mongod : 6.0.6
  Elasticsearch : 7.10.2

• Service logs, configurations, and environment variables:
  Graylog :
  index [linux_year_48], type [_doc], id [], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index can't exceed [2147483519]

Elasticsearch when I try to rotate index :

{"type": "server", "timestamp": "", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "", "node.name": "", "message": "adding template [linux_year-template] for index patterns [linux_year_*]", "cluster.uuid": "", "node.id": ""  }

3. What steps have you already taken to try and solve the problem?
I’ve tried to rotate index on Graylog
I’ve tried to create a new index set but it’s not working
I’ve tried to rotate index through Elasticseatch CLI but I didn’t find out how to do

4. How can the community help?
I’m searching to fix this issue, I need index rotation, creation and deletion to work.
Can you help me for that case please ?

Best regards.

[drewmiranda-gl (Drew Miranda)]

Can you share your index template settings and rotation settings for linux_year_ index set? Via System/Indices

Also can you confirm if this message is correct or not?

Number of documents in the index can’t exceed [2147483519]

Do you have that many messages in a single index?

Also are you using multiple index sets and do all of them have this same issue, or are you only using a single index set?

[VirtualFloppy]

Hello,

Thank you for your answer
I’m sorry but as new user on this forum I cannot send any pictures.

Please find below index set configuration :

• 15 days rotation
• deletion after 26 index rotations
• 1 shard
• 1 replica

I confirm that the message is this one : Number of documents in the index can’t exceed [2147483519].

I also confirm that I also have a lot of messages in the last index because rotation is not working anymore. I’m speaking about 1,5To messages for this last index made of 2147483519 messages.

I am using multiple index sets (27) and all have the same issues since Graylog have been updated to version 5 (I didn’t check each but some of them). The others are not full of messages, but there is no more index rotation.

I’ve made a bash script that allow me to output indexes name, age in days and index timestamp from Elasticsearch. There, I can see that before May 17th rotation was working ; one rotation each 15 days as configured. And then, after the upgrade, no more rotation : current index is always the 48th which have May the 17th as timestamp.

When I try to force rotation through Graylog “Rotate active write index” button (system > indices > my_index and top right button Maintenance) nothing happens, except the same message on Elasicsearch logs :

{"type": "server", "timestamp": "", "level": "INFO", "component": "o.e.c.m.MetadataIndexTemplateService", "cluster.name": "", "node.name": "", "message": "adding template [linux_year-template] for index patterns [linux_year_*]", "cluster.uuid": "", "node.id": ""  }

When I try to create a new index set it’s not working.

It’s like if Graylog has no control on Elasticsearch index management anymore.
I’m thinking about rollback to previous version, but I don’t want to stay in older versions.

Thank you for your help.

Best regards.

[drewmiranda-gl (Drew Miranda)]

Are you able to share the last 100-200 lines of your graylog server.log as well as the elasticsearch server log?

I’m very curious!

If they are too long you can use something like https://pastebin.com/ .

Thanks!

[VirtualFloppy]

I’ve searched further when exporting logs for you and I found the issue with that log message : this action would add [2] total shards, but this cluster currently has [3056]/[3000] maximum shards open;.
This log message was drowned in these very numerous other messages : Number of documents in the index can’t exceed [2147483519].

I’ve increesed the number of allowed shards : curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'
I also deleted 365 days old shards.

Thank you for your help, this case is solved !!

Best regards.