Unprocessed messages during index optimization

Graylog Central (peer support)
1

Graylog Version: 2.4.5
Elastic Version: 5.6.9

Graylog Nodes: 8 CPUs, 32 MB RAM
Elastic Nodes: 4 CPU, 64 MB RAM
Indicies: 97, 1,816,743,448 documents, 1.7TB

We have a cluster with 2 Graylog and 4 Elastic Nodes. During index optimization, the number of unprocessed messages increases dramatically and the elastic nodes seem to slow indexing messages too much. Does anyone know a solution for this issue or have experience with customizing the index.merge.scheduler.throttle value in Elastic?

2

you might want to lower the refresh.index setting in your index template to something around 30seconds. This would be one option to lower the pain on elasticsearch.

regards
Jan

3

Hi Jan,

thanks.

I assume you mean the refresh_interval as described here

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-update-settings.html.

That would mean I would have to increase the interval with a cron job before optimizing and then set to default again.

Changing the index.merge.scheduler.throttle value is not a sensible permanent option ?

regards
Jürgen

4

He Jürgen,

you can do that, but with that I didn’t experiment much to have a valid opinion on that. Only the first one is something I do regular.

jan

5

ok, then i will first test your suggestion

thx

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.