Graylog Data Tiering does not create a new index and the active write index gets index.blocks.write

[Ascalon (Denis Didkovskiy)]

1. Describe your incident:

Hello,

I am troubleshooting an issue with Data Tiering in Graylog.

My index set uses:

• Rotation strategy: Data Tiering
• Min. in storage: 1 day
• Max. in storage: 21 days

The problem is that Graylog does not create a new index.
The same index remains the active write index, and eventually OpenSearch sets index.blocks.write on it.

Because of that, writes are blocked instead of Graylog rotating to a new write index.

2. Describe your environment:

• Graylog version: 6.3.4
• OpenSearch version: 2.19.4

3. What steps have you already taken to try and solve the problem?

• I checked the index set configuration in Graylog.
• I verified that the rotation strategy is Data Tiering.
• I observed that no new index is created.
• In OpenSearch, the active write index eventually gets index.blocks.write.

4. How can the community help?

Could someone please clarify:

• What exactly triggers creation of a new write index when using Data Tiering?
• Should Graylog create a new index automatically before the current one becomes write-blocked?
• Can Data Tiering rotate indices by age alone, or does it also depend on other conditions?
• Is index.blocks.write expected in this scenario, or does it indicate a separate OpenSearch-side issue?

At the moment, the main symptom is:
Graylog does not rotate to a new index, and the current write index becomes blocked for writes in OpenSearch.

If needed, I can also provide screenshots of the index set configuration and OpenSearch index settings.

Thank you.

[Wine_Merchant]

Hey @Ascalon,

Are there logs from the Graylog server and Data Node server that can offer more insight into what is occurring at the point of rotation?

Has this just started happening or only occurring after switching to a new rotation strategy?

• What exactly triggers creation of a new write index when using Data Tiering?
  It is either age or size, there is a default shard size that Graylog will try to aim for. The index will try to get as close to that shard size before rotating and if it reaches the shard size then rotation occurs.

• Should Graylog create a new index automatically before the current one becomes write-blocked?
  A new index becomes the active write index and at that point the prior index becomes read only.

• Can Data Tiering rotate indices by age alone, or does it also depend on other conditions?
  As mentioned above, it is either age or size

• Is index.blocks.write expected in this scenario, or does it indicate a separate OpenSearch-side issue?
  Is a new index not being created at the point of rotation?