Problem with unprocessed messages in inputs and index

[omm79 (oscar)]

Hi, my configuration is:
Linux Ubuntu 18.04
Graylog 2.4.4+4659dbe, codename Wildwuchs
Elasticsearch “number” : “5.6.9”
mongodb version v3.4.14
only 1 node

at this point I’ve graylog_9 index generated and it shows 4 shards green.
Changing the number of replicas from 0 to 2 and viceversa

my problem is that everything was fine until 1 month ago but graylog stop processing messages in inputs and i’ve tried almost everything in this site: https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/#reason-1-shard-allocation-is-purposefully-delayed

I’ve already restart elasticsearch service with systemctl, I don’t want to loose index information that still in the memory or journal so I haven’t restarted graylog and mongodb services, only restarted elasticsearch, also elasticsearch and graylog logs don’t show any problem about it.

Also I tried some parameters adjustments to generate slowlog in the index graylog_9 at node level:

curl -XPUT “localhost:9200/_settings?pretty” -H ‘Content-Type: application/json’ -d’{“index.search.slowlog.threshold.query.warn”: “5s”, “index.search.slowlog.threshold.query.info”: “5s”, “index.search.slowlog.threshold.query.debug”: “2s”, “index.search.slowlog.threshold.query.trace”: “500ms”, “index.search.slowlog.threshold.fetch.warn”: “1s”, “index.search.slowlog.threshold.fetch.info”: “800ms”, “index.search.slowlog.threshold.fetch.debug”: “500ms”, “index.search.slowlog.threshold.fetch.trace”: “200ms”, “index.search.slowlog.level”: “debug”, “index.indexing.slowlog.threshold.index.warn”: “5s”}’

curl -XPUT “localhost:9200/_settings?pretty” -H ‘Content-Type: application/json’ -d’{“index.indexing.slowlog.threshold.index.warn”: “10s”, “index.indexing.slowlog.threshold.index.info”: “5s”, “index.indexing.slowlog.threshold.index.debug”: “2s”, “index.indexing.slowlog.threshold.index.trace”: “500ms”}’

curl -XPUT -d ‘{“index.search.slowlog.threshold.query.warn” : “50ms”,“index.search.slowlog.threshold.fetch.warn”: “50ms”,“index.indexing.slowlog.threshold.index.warn”: “50ms”}’ http://127.0.0.1:9200/graylog_9/_settings | jq

and still the same it’s not processing messages to the index, I hope you can help me to solve this, because at some point my node will fail, thanks.

[jan (Jan Doberstein)]

he @omm79

how much data did you have in that elasticsearch? How much RAM did you have on that server and how much HEAP did you configured for Elasticsearch? Did you have all on one server? If yes, how much HEAP have you configured for Graylog?

How much of your disk space is used?

The description is likely that you hit the limits of elasticsearch in your environment and need to lower the amount of data or raise some of limiting factors.

[omm79 (oscar)]

I couldn’t upload more images, so I decided restart the server but I lost 1 month information :S, even so there were unprocessed messages and I rotate the index and I think It worked, there aren’t more unprocessed messages and the new index is storing the messages so there’s nothing to do to recover the information lost, thanks for your reply, learned lesson to check if there aren’t unprocessed messages and they are stored in the index:

my specific problem is that since the begining I notice that in the directory from logs graylog and elasticsearch (/var/log/elasticsearch) didn’t create graylogprod-DATE.log and graylogprod_index_indexing_slowlog-DATE.log logs but before I rotate the index they did it, so I tried to configure parameters like I said in the first post but I think those changes hang the index, so my question now is What I have to configure o what CURL code I have to put to configure the slowlogs and logs to the new index without hang it?, thanks for your help, regards.

[omm79 (oscar)]

some others configurations:

its all on 1 server, 1 node no replication, I’ll wait for your recommendations, thanks.

[jan (Jan Doberstein)]

if you have everything on one server - what is the configured log rotation and retention stragegy?

With the default journal configuration you have not more than 8GB elasticsearch storage data left on your server I guess. Did you honored that in the index rotation and retention settings?

Jan