Elasticsearch nodes disk usage above low watermark

Graylog Central (peer support)
1

1. Describe your incident:
In System/Overview of my freshly installed Graylog version 6.1.1 (there’s nothing configured, no data), I have notification “Elasticsearch nodes disk usage above low watermark”.

I don’t understand why I get this warning because my instance is using Data Node and services like ‘elasticsearch’ or ‘opensearch’ I don’t have installed and don’t see them running.

2. Describe your environment:

  • OS Information:
    I’m using latest version of Debian 12 Bookworm with Graylog version 6.1.1

Can someone explain how to get a rid of this warning, does it mean any problem, are there any solutions to this?

Thanks everyone for suggestions and have great day everyone.

2

Hello @manoli,

It means that there isn’t enough space to write new messages to Opensearch, the DataNode by default writes to /var/lib/graylog-datanode. Did you add a drive with enough space here when configuring your Docker instance?

3

Hi and thanks for reply.

I have to check Data Node configuration, which path it’s actually writing to…

But I’m not using Docker.

It’s Debian based VM initially with system disk 20G allocated and there’s nothing else on it so, I will check, but I believe there must be like 16G free (available).

UPDATE:

Yes I just checked the directory is present at /var/lib/graylog-datanode/ but it’s currently using 30M only.

VM’s root volume / have 20G allocated used by 33% it means exactly 13G is still available. (Is this not enough, could this be reason triggering the warning?)

4

I should have mentioned that DataNode is a wrapper around Opensearch so technically you are using Opensearch. The reference to Elasticsearch in the error is legacy.

I would expect to see this issue if the data store had utilised around 85% of the storage so it’s not clear why this is happening.

Please check the logs of the DataNode to get a better idea what exactly is occurring, post them back here if something isn’t clear.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

6

There are additional 10G allocated for the searchable snapshots cache.

This means you actually have only 3G free and this triggers watermark warnings.

If you are not using data tiering, then you can significantly reduce the cache size. Try the following setting in your graylog configuration file:

node_search_cache_size: 1g

This should free additional 9G of space on your disk and stop the watermark warnings.

Best regards,
Tomas