Data node configuration override not taking effect

Graylog Central (peer support)
1

1. Describe your incident:

Recently I migrated from self-managed Opensearch to Graylog Data Node. In Opensearch we used to have custom disk watermark settings that are higher than the default. After migration however it seems that these settings were lost as they were set through API not a configuration file.

I am trying to set the same watermark settings for Data Nodes. Since I don’t have direct access to the API any more I figured the best way to achieve this was to use Data Node configuration override:

I followed the instructions: add override file location to datanode.conf, create override file with following settings:

cluster.routing.allocation.disk.watermark.low = 95%
cluster.routing.allocation.disk.watermark.high = 97%
cluster.routing.allocation.disk.watermark.flood_stage = 99%

Did this on all 4 data nodes and 3 master nodes and then did a rolling restart on the whole cluster. The settings seem to take effect on each node individually as they boot up but as soon as they rejoin the cluster it seems like the cluster defaults are forced thus changing the values back to default again.

2. Describe your environment:

4 graylog data nodes, 3 cluster manager nodes

  • OS Information: Debian 13

  • Package Version: 7.1.8-1

  • Service logs, configurations, and environment variables:

2026-09-29T16:20:21.351+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:21,350][INFO ][o.o.n.Node               ] [os-data-04.domain.redacted] starting ...
2026-09-29T16:20:21.504+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:21,504][INFO ][o.o.t.TransportService   ] [os-data-04.domain.redacted] publish_address {os-data-04.domain.redacted/ip-address}, bound_addresses {ip-address}
2026-09-29T16:20:23.819+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:23,818][INFO ][o.o.b.BootstrapChecks    ] [os-data-04.domain.redacted] bound or publishing to a non-loopback address, enforcing bootstrap checks
2026-09-29T16:20:23.825+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:23,825][INFO ][o.o.c.c.Coordinator      ] [os-data-04.domain.redacted] cluster UUID [3DgCW21xRk-l6Oqwe0-SwQ]
2026-09-29T16:20:26.332+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,332][INFO ][o.o.c.s.ClusterApplierService] [os-data-04.domain.redacted] cluster-manager node changed {previous [], current [{os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true}]}, added {{os-master-03.domain.redacted}{28gvXXrxSrWB6bzkv3MGwA}{QQsBaP_bR3efjLJgBjP2zQ}{os-master-03.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true},{os-master-02.domain.redacted}{uc0pPfrtQKavzhQgOjERzw}{sO_NIsNhQb6QzQFGf9Y60A}{os-master-02.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true},{os-data-03.domain.redacted}{VmBGMHhETPeovaMCKvB6YQ}{xPkABkOYR0OJ6-SaIw8PfQ}{os-data-03.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-data-02.domain.redacted}{iIwcH22cTC2Wvouj459mIw}{d6OuWakqQM-cfUePyCQBgA}{os-data-02.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-data-01.domain.redacted}{dJH-nBD1SsmOSmhr1UQ5BA}{cHE2Vex7R8SIg7KT7WPQTw}{os-data-01.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true}}, term: 195, version: 383130, reason: ApplyCommitRequest{term=195, version=383130, sourceNode={os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}}
2026-09-29T16:20:26.534+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [cluster.info.update.interval] from [30s] to [1m]
2026-09-29T16:20:26.534+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.low] from [95%] to [90%]
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.high] from [97%] to [95%]
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [plugins.index_state_management.metadata_migration.status] from [0] to [1]
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO ][o.o.i.i.ManagedIndexCoordinator] [os-data-04.domain.redacted] Canceling metadata moving job because of cluster setting update.
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.low] from [95%] to [90%]
2026-09-29T16:20:26.535+03:00 INFO  [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO ][o.o.c.s.ClusterSettings  ] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.high] from [97%] to [95%]

3. What steps have you already taken to try and solve the problem?

Reread the documentation, relevant Opensearch documentation, looked for similar topics.

4. How can the community help?

Am I doing something wrong? Or is there some bug in data node configuration override? I know there is a warning on the Data Node Configuration Override documentation page that says the overrides are not fully supported by Graylog. Perhaps it’s not working as intended?

2

I was able to get around this by getting access to the API by generating a client certificate in the UI and adding it to admin_dn as described in this post:

With this cert I am once again able to access the API and used it to set the watermarks via API:

curl -sk   --cert /etc/graylog/datanode/client-cert.crt   --key /etc/graylog/datanode/client-cert.key   -X PUT   'https://os-master-01.domain.redacted:9200/_cluster/settings'   -H 'Content-Type: application/json'   -d '{
    "persistent": {
      "cluster.routing.allocation.disk.watermark.low": "95%",
      "cluster.routing.allocation.disk.watermark.high": "97%",
      "cluster.routing.allocation.disk.watermark.flood_stage": "99%"
    }
  }'

curl -sk   --cert /etc/graylog/datanode/client-cert.crt   --key /etc/graylog/datanode/client-cert.key   'https://os-master-01.domain.redacted:9200/_cluster/settings?pretty'
{
  "persistent" : {
    "cluster" : {
      "routing" : {
        "rebalance" : {
          "enable" : "all"
        },
        "allocation" : {
          "disk" : {
            "threshold_enabled" : "true",
            "watermark" : {
              "low" : "95%",
              "flood_stage" : "99%",
              "high" : "97%"
            }
          },
          "enable" : "all"
        }
      },
      "info" : {
        "update" : {
          "interval" : "1m"
        }
      }
    },
    "opendistro" : {
      "index_state_management" : {
        "history" : {
          "number_of_replicas" : "1"
        }
      }
    },
    "plugins" : {
      "index_state_management" : {
        "metadata_migration" : {
          "status" : "1"
        },
        "template_migration" : {
          "control" : "-1"
        }
      }
    }
  },
  "transient" : { }
}