Disk full because of elasticsearch

[tor]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I have an warning saying “elasticsearch nodes disk usage above high watermark”.
In /var/lib/elasticsearch/nodes/0/indies I have 800G of data.

2. Describe your environment:

• OS Information: Centos 7.9

• Package Version: Graylog 4.2.1

3. What steps have you already taken to try and solve the problem?
None not sure what I can do to fix this. All logs have stopped in graylog

[tmacgbay]

There are many posts in the community about how to clear high watermarks from Elasticsearch… here is an Elasticsearch article that addresses some of the issue as well You didn’t post what version of elasticsearch you are on so I chose randomly to get you started. You can also find people asking and getting replies on how to move data out (archive) of Elasticsearch or out on the internet on how to properly increase disk spaces based on your configuration. Use the search, Luke!

[tmacgbay]

Here is a more in depth post by @gsmith that talks about watermarks and what to do with them:

Graylog stopped with timeout

Hello, My apologies, I misread your post. I believe Its because of this statement in your logs. low disk watermark [85%] exceeded on Definition of low disk watermark [85%] exceeded. When disk usage on a host hits 85 percent, the Elasticsearch service prevents shard allocation and stops working. This disk usage threshold is an Elasticsearch configuration. By default, the cluster.routing.allocation.disk.watermark.low watermark is set to 85% to prevent Elasticsearch from allocating new sh…

[tor]

Thanks. I increased my disk space as I knew the partition for elasticsearch for over 90% full. Good thing is a vm I just extended the partition. All good now.