Unprocessed messages during index optimization

Graylog Version: 2.4.5
Elastic Version: 5.6.9

Graylog Nodes: 8 CPUs, 32 MB RAM
Elastic Nodes: 4 CPU, 64 MB RAM
Indicies: 97, 1,816,743,448 documents, 1.7TB

We have a cluster with 2 Graylog and 4 Elastic Nodes. During index optimization, the number of unprocessed messages increases dramatically and the elastic nodes seem to slow indexing messages too much. Does anyone know a solution for this issue or have experience with customizing the index.merge.scheduler.throttle value in Elastic?

you might want to lower the refresh.index setting in your index template to something around 30seconds. This would be one option to lower the pain on elasticsearch.


Hi Jan,


I assume you mean the refresh_interval as described here


That would mean I would have to increase the interval with a cron job before optimizing and then set to default again.

Changing the index.merge.scheduler.throttle value is not a sensible permanent option ?


He Jürgen,

you can do that, but with that I didn’t experiment much to have a valid opinion on that. Only the first one is something I do regular.


ok, then i will first test your suggestion


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.