Unprocessed messages during index optimization


Graylog Version: 2.4.5
Elastic Version: 5.6.9

Graylog Nodes: 8 CPUs, 32 MB RAM
Elastic Nodes: 4 CPU, 64 MB RAM
Indicies: 97, 1,816,743,448 documents, 1.7TB

We have a cluster with 2 Graylog and 4 Elastic Nodes. During index optimization, the number of unprocessed messages increases dramatically and the elastic nodes seem to slow indexing messages too much. Does anyone know a solution for this issue or have experience with customizing the index.merge.scheduler.throttle value in Elastic?

(Jan Doberstein) #2

you might want to lower the refresh.index setting in your index template to something around 30seconds. This would be one option to lower the pain on elasticsearch.



Hi Jan,


I assume you mean the refresh_interval as described here


That would mean I would have to increase the interval with a cron job before optimizing and then set to default again.

Changing the index.merge.scheduler.throttle value is not a sensible permanent option ?


(Jan Doberstein) #4

He Jürgen,

you can do that, but with that I didn’t experiment much to have a valid opinion on that. Only the first one is something I do regular.



ok, then i will first test your suggestion


(system) closed #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.