Recurring error every 30 seconds: [IndexerClusterCheckerThread] - Error while trying to check Elasticsearch disk usage

Graylog Central (peer support)
1

Every 30 seconds on the dot we are getting the following error:

2023-06-29 20:07:29,248 ERROR [IndexerClusterCheckerThread] - Error while trying to check Elasticsearch disk usage.Details: Error creating ClusterAllocationDiskWatermarkSettings. This should never happen. - {}

We have checked disk space, recreated all pods, etc. All indexes have been rotated. We searched google for similar issues, but nothing seems to apply. Any ideas on where to look?

Environment:
Kubernetes/docker running pods on Ubuntu 22.04
Graylog version: 5.1.2+d970230
ElasticSearch: 7.10.2
MongoDB: 6.0.6

2

Hey @amunoz

Has this just started? If so what happened before this issue occurred.

ES maybe in read mode only.

3

Thank you for the response @gsmith!

We only keep 90 days of logs and it’s in all 90 days so we’re not able to determine when it started unfortunately. Read-only mode is not enabled. Any other suggestions on where to look are much appreciated!

4

Hey @amunoz

I seen something else familiar while back

5

Thanks for the link. Looks like a different issue unfortunately as we are on ES 7.10.2. Also of note, this only happens in production, not in dev. They are both using the same versions of everything.

6

Hey @amunoz

what do you see when executing this?

curl -XGET http://localhost:9200/_cat/allocation?v&pretty
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.