So. Elasticsearch 6.7 released ILM, which helps you manage your indexes and can perform a variety of tasks (force merge, shrink, allocation to different class of node) automatically based on a policy you define.
If you want to make use of it on your Graylog cluster, you can, but there’s a few things you have to do:
1: Set your indexes to time-based rotation; because the ILM policies trigger on size, index age, or document count. If both Graylog and ILM trigger on an index (size, count), you’re in for a bad time, so you need time based rotation. I personally use P1D.
2: Set your initial “warm” phase time to be at least 36 hours from index creation to make sure Graylog has cycled it.
3: Disable the index optimization after rotation in Graylog and let ILM do it.
4: Read up on “Frozen” indices, and make sure people are aware that searches past a certain time range (if you use the index freeze option) can be slow as dirt.
Ideally… and I do mean ideally… we’d get an option in Graylog to disable it’s index rotation mechanisms in favor of setting up a simple roll-over in ES itself, because then the ILM stuff can be used to it’s full extent.
I’m not aware if you could set an index alias with the ILM and if the ILM can make a API call to Graylog if the job is done?
Backround: Graylog uses the deflector ( http://docs.graylog.org/en/3.0/pages/configuration/index_model.html ) and needs to get some kind of ‘notification’ if the indices are manipulated from 3rd party. That is what happens with the “recalculate index range” - Graylog checks the indice sets it has configured what indices are available with the name pattern and make a min/max query to know what timerange is in this index available.
Short answer - you can use the noop function to rotate only with Graylog, but making this optional in Graylog does not work.
ILM can handle rollover of an index for you, but it does require some preliminary setup (here) and then a policy that applies the terms for the rollover (here)
The way I would like to see it work is that Graylog (when you create a new index set) sets up the initial index + alias, then generates an ILM policy based on the settings you make (perhaps extended with the whole warm/cold phase and the ability to set allocations, as well as force merge/shrink/freeze options) and then lets ES handle all the rotation/moving/optimizing. ES also takes care of moving the alias around, when a rollover index is set up.
I know it’s a bit of a stretch to make that work, but maybe if the index management (rotation, creation, etc.) is abstracted to a plugin architecture we can have it as a plugin of sorts. I guess similarly to the retention personalities.
Anyway! The way I have this working now is that I let Graylog do the rotation of the index (without optimizing), and ILM kicks in after to optimize it, and to set the allocations. (The reason for my using ILM is that we recently moved to a hot/warm/frozen setup, we have 3 beefy SSD based hot servers for indexing and holding the “current” write indices, that then get moved to the warm machines and optimized, and then moved to the frozen ones (40Tb capacity, go go gadget disks), where they get frozen and kept for, well, not sure yet, A year+ at the least - or whenever disk space runs low.
