So. Elasticsearch 6.7 released ILM, which helps you manage your indexes and can perform a variety of tasks (force merge, shrink, allocation to different class of node) automatically based on a policy you define.
If you want to make use of it on your Graylog cluster, you can, but there’s a few things you have to do:
1: Set your indexes to time-based rotation; because the ILM policies trigger on size, index age, or document count. If both Graylog and ILM trigger on an index (size, count), you’re in for a bad time, so you need time based rotation. I personally use P1D.
2: Set your initial “warm” phase time to be at least 36 hours from index creation to make sure Graylog has cycled it.
3: Disable the index optimization after rotation in Graylog and let ILM do it.
4: Read up on “Frozen” indices, and make sure people are aware that searches past a certain time range (if you use the index freeze option) can be slow as dirt.
Ideally… and I do mean ideally… we’d get an option in Graylog to disable it’s index rotation mechanisms in favor of setting up a simple roll-over in ES itself, because then the ILM stuff can be used to it’s full extent.