Log Output Constantly Stalling to Zero

Graylog Central (peer support)
3

@jan through the Graylog GUI I changed within the “Default index set” the “Field type refresh interval” value to 30 seconds (it was previously set to 5). I found reference to what this is (I think) at What is the parameter "field_type_refresh_interval"? and unless I am misunderstanding, it supposedly correlates to the “refresh_interval” in Elasticsearch. I changed it and let the index rotate but after checking the newest index, I didn’t see “refresh_interval” set to 30 seconds as expected. Not sure if I should change that back to 5 seconds or if it matters at all.

Since the previous step didn’t seem to do what I expected, I went ahead and set it within Elasticsearch for the default template:

# curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_template/all_indices -d '{"template":"*","settings":{"refresh_interval":"30s"}}'

I waited for the index to rotate and checked that the newest index had the “refresh_interval” set to 30 seconds:

# curl --silent http://localhost:9200/graylog_*?pretty | cut -c4-16 | grep ^graylog_ | while read INDEX; do curl --silent http://localhost:9200/${INDEX}/_settings?pretty; done | grep "graylog_\|refresh_interval" | grep -v provided_name | tail -2
  "graylog_99233" : {
        "refresh_interval" : "30s",

Not sure if it was necessary but I also looped through all existing Graylog indices and set the existing ones to 30 seconds also:

# curl --silent http://localhost:9200/graylog_*?pretty | cut -c4-16 | grep ^graylog_ | while read INDEX; do curl --silent -XPUT -H 'Content-Type: application/json' http://localhost:9200/${INDEX}/_settings -d '{"index":{"refresh_interval":"30s"}}'; done

And verified that all existing Graylog indices were now set to 30 seconds for the “refresh_interval” setting:

# curl --silent http://localhost:9200/graylog_*?pretty | cut -c4-16 | grep ^graylog_ | while read INDEX; do curl --silent http://localhost:9200/${INDEX}/_settings?pretty; done | grep "graylog_\|refresh_interval" | grep -v provided_name
  "graylog_99052" : {
        "refresh_interval" : "30s",
  "graylog_99053" : {
        "refresh_interval" : "30s",
  "graylog_99054" : {
        "refresh_interval" : "30s",
[[removed additional output]]

I also remember several months ago trying to fix a similar (same?) performance issue and tinkering with the “output_batch_size” setting in the “graylog.conf” configuration file. I had it set to 10000 (seems way high) and found a Github post where a user had set it to 1000 with positive results and later in that thread I saw a developer comment that the new default had been set to 500. I tried 500 at first and it did not help at all. I am now running it at 1000 instead of the 10000 I had it initially set at but it still isn’t helping.

As for taking “a deeper look into the threads and information” within Elasticsearch, with me being an Elasticsearch novice, might you have some advice on specifically what I should be looking for? When Google searching around I found reference to “/_cat/thread_pool” which I queried but I am not sure exactly what I should be looking for that would be indicative of a bottleneck or problem or if this is even the right thing you’re talking about.

I also found reference in my Google search to “/_nodes/hot_threads” which I also queried, but again, same as above, not sure what I should be looking for that is indicative of a bottleneck or problem or if this is even what you’re asking me to check into.

Any advice is greatly appreciated.