We’re upgrading from Graylog Open 5.2 → 6.2 and I’m trying to wrap my head around the new index rotation options. Looks like we’re leapfrogging the now-deprecated “Index Time Size Optimizing” config and the recommended way of handling index rotation is via data tiering.
The data tiering docs seem to only apply to Enterprise. In our case, we’re handling data tiering natively in opensearch with ISM policies. What’s the recommended index rotation configuration for graylog open? Should we still use the deprecated index rotation model, and if so, how long can we expect that to stick around? I don’t like to use deprecated features and would prefer to do what yall recommend, but it’s not clear from the documentation what path to take.
Hi @brosef, for Graylog open you can also use data tiering, just without warm tier and archiving. For example, if you have the following “Index Time Size Optimizing” configuration:
Also what can I expect when upgrading? We’re planning on a rolling upgrade, so there will be a span of time with mixed 5.2 and 6.2 members. After the upgrade do I need to go into the UI and point-and-click change the index rotation settings?
Nope, that is 100% GUI now. Data Tiering without enterprise is pretty much the same as the “old” legacy dynamic rotation.
When you do the upgrade the new options will be available, but stay with your old default values.
For the time your cluster is not on the same version I would recommend to stay with the old option which is available for all nodes. I would try to make that time as short as possible and avoid a mixed version environment.
yes, that’s how we’ve been handling index rotation. I’m just disappointed by:
a) configuration is now GUI-only, so we have no change control over these config changes (i.e. git PR), as well as no automatic way of deploying configuration. If I don’t like the defaults, I have to log in to the app and manually update the configuration. In the DevOps world, that’s an antipattern.
b) IMO this was a pretty significant change and I don’t see anything in the release notes about it, no guidance on how to migrate from the legacy stuff, no mention of moving away from server.conf (as far as I can tell). Generally, the documentation around the new index model is confusing for those of us using Open. Is this accurate? - Index Defaults. I see no mention of Data Tiering, but this is called out - " Unless user-specified defaults are configured, the following defaults will be effective for all new index sets created: [Shards: 1] [Rotation Strategy: Time Size Optimizing - 30-40 days]"
