Graylog server+datanode 7.0.11 Watermark calculation incorrect/unclear

Graylog Central (peer support)
1

1. Describe your incident:

After migrating our Graylog instance to a new server (as a part of widespread OS replacement process) we started encountering frequent Flood Watermark warnings that keep coming back.

Importantly, even though we get these flood stage warnings, cluster still receives and stores data (disk usage grows as usual, indexes have expected sizes after rotations, we don’t seem to lose any data). As I understand the situation, when disk does fill up, we won’t have any advance warning that we do start dropping data.

  • OS Information: Ubuntu 22.04.5 LTS

  • Package Version: graylog-server 7.0.11-1, graylog-datanode 7.0.11-1

It seems that even though we set our custom values in opensearch_configuration_overrides_file:

cluster.routing.allocation.disk.watermark.low = 15gb cluster.routing.allocation.disk.watermark.high = 10gb cluster.routing.allocation.disk.watermark.flood_stage = 5gb

These values are not taken into account.
Checking values via OpenSearch API shows what we want:

"disk" : {
  "threshold_enabled" : "true",
  "watermark" : {
    "flood_stage" : "5gb",
    "high" : "10gb",
    "low" : "15gb",
    "enable_for_single_data_node" : "false"

It seems that we have enough disk space:

Filesystem                         1M-blocks     Used Available Use% Mounted on
/dev/mapper/vg_graylog-lv_datanode  1243571M 1165118M    75910M  94% /opt/graylog-datanode

Reserved blocks amount to ~2.5GB:

Block count:              323464192
Reserved block count:     646928
Free blocks:              21068949
Block size:               4096

And yet we still seem to get the watermark triggered without apparent relation to actual free disk space:

image
image2342×285 35.4 KB

2026-07-23T18:45:44.605+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-23T19:14:44.572+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-23T19:57:44.599+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-23T20:26:44.569+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-23T20:43:14.588+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-23T20:59:14.581+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T02:04:44.595+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T10:49:44.617+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T11:03:44.574+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T11:30:14.595+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T11:34:14.600+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T11:53:14.589+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-24T12:02:44.586+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-25T02:02:14.622+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-25T02:37:14.605+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-25T02:53:44.588+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-25T03:11:44.571+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-25T03:28:44.583+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-26T05:41:14.588+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-26T06:05:44.574+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-26T07:00:14.589+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-26T07:17:44.574+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-26T07:47:14.591+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T02:13:44.587+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T02:44:44.602+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T07:21:14.601+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T07:43:44.589+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T07:55:14.589+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T07:56:14.602+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T08:09:14.616+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T08:25:14.610+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T08:33:44.596+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-27T09:32:44.578+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T02:01:44.612+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T02:20:14.597+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T02:36:14.600+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T02:54:44.588+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T03:11:44.602+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-28T03:28:14.612+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-07-31T08:14:24.714+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-08-04T13:24:51.519+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space
2026-08-05T12:02:51.521+02:00 WARN  [IndexerClusterCheckerThread] Elasticsearch node [ip.a.d.dr] triggered [ES_NODE_DISK_WATERMARK_FLOOD_STAGE] due to low free disk space

Does that seem to be a bug in Graylog Server? What else should I verify to better understand our situation?

Thanks for any suggestions!
Wojciech

2

Hi @roman_the ,

Just to be sure, this custom overrides are applied to each of your datanodes, right? Another thing I’d check is that you don’t point your datanodes to multiple data dirs at once?

You already verified that the opensearch API is responding correctly with your set values, so if that’s true for all datanodes, we can safely say that the datanode part is fine.

You can also check what opensearch sees under `_cat/nodes?v&h=name,ip,disk.total,disk.used,disk.avail,disk.used_percent`.

I double-checked the graylog server logic for triggering the notification and haven’t found any obvious bug. And since it started happening after the migration itself, I’d suspect the datanode configuration first.

One more idea - are you using warm tier / snapshots? Do you have `node_search_cache_size` configured for your datanodes? If yes, how?

3

Hello Tomas,

thanks for your reply, let me answer your questions below:

  1. There is only single datanode in our instance currently, so that’d be a yes :wink:

2. Data directory - default config in /etc, /var/lib/graylog-datanode is symlinked to a second partition. Root filesystem has 35% (3.3GB) available, Opensearch partition has 4% (58GB) available.

3. API call output, with hostname and IP edited:
name ip disk.total disk.used disk.avail disk.used_percent
HOSTNAME IP 1.1tb 1.1tb 57.6gb 95.25

  1. Snapshots and warm tier - we’re not using either. Snapshots were used for the migration, but I’ve removed them right after switchover.

5. node_search_cache_size - I don’t have this parameter configured.

Best regards,
Wojciech