# Data node configuration override not taking effect **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** data-node\ **Created:** [October 5, 2026, 8:01am UTC](https://community.graylog.org/t/data-node-configuration-override-not-taking-effect/37617 "2026-10-05T08:01:16Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![ovreba](https://avatars.discourse-cdn.com/v4/letter/o/bc79bd/32.png) [@ovreba](https://community.graylog.org/u/ovreba)\ **Post date:** [October 5, 2026, 8:01am UTC](https://community.graylog.org/t/data-node-configuration-override-not-taking-effect/37617/1 "2026-10-05T08:01:16Z")
**1. Describe your incident:** Recently I migrated from self-managed Opensearch to Graylog Data Node. In Opensearch we used to have custom disk watermark settings that are higher than the default. After migration however it seems that these settings were lost as they were set through API not a configuration file. I am trying to set the same watermark settings for Data Nodes. Since I don’t have direct access to the API any more I figured the best way to achieve this was to use Data Node configuration override: > **[Data Node Configuration Overrides](https://go2docs.graylog.org/current/setting_up_graylog/data_node_configuration_overrides.htm)** > > Learn how to configure override settings for Graylog Data Node using custom OpenSearch configuration files. Follow best practices, warnings, and supported allowlist parameters to enhance your setup. I followed the instructions: add override file location to datanode.conf, create override file with following settings: cluster.routing.allocation.disk.watermark.low = 95% cluster.routing.allocation.disk.watermark.high = 97% cluster.routing.allocation.disk.watermark.flood\_stage = 99% Did this on all 4 data nodes and 3 master nodes and then did a rolling restart on the whole cluster. The settings seem to take effect on each node individually as they boot up but as soon as they rejoin the cluster it seems like the cluster defaults are forced thus changing the values back to default again. **2. Describe your environment:** 4 graylog data nodes, 3 cluster manager nodes - OS Information: Debian 13 - Package Version: 7.1.8-1 - Service logs, configurations, and environment variables: ```auto 2026-09-29T16:20:21.351+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:21,350][INFO][o.o.n.Node] [os-data-04.domain.redacted] starting ... 2026-09-29T16:20:21.504+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:21,504][INFO][o.o.t.TransportService] [os-data-04.domain.redacted] publish_address {os-data-04.domain.redacted/ip-address}, bound_addresses {ip-address} 2026-09-29T16:20:23.819+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:23,818][INFO][o.o.b.BootstrapChecks] [os-data-04.domain.redacted] bound or publishing to a non-loopback address, enforcing bootstrap checks 2026-09-29T16:20:23.825+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:23,825][INFO][o.o.c.c.Coordinator] [os-data-04.domain.redacted] cluster UUID [3DgCW21xRk-l6Oqwe0-SwQ] 2026-09-29T16:20:26.332+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,332][INFO][o.o.c.s.ClusterApplierService] [os-data-04.domain.redacted] cluster-manager node changed {previous [], current [{os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true}]}, added {{os-master-03.domain.redacted}{28gvXXrxSrWB6bzkv3MGwA}{QQsBaP_bR3efjLJgBjP2zQ}{os-master-03.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true},{os-master-02.domain.redacted}{uc0pPfrtQKavzhQgOjERzw}{sO_NIsNhQb6QzQFGf9Y60A}{os-master-02.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true},{os-data-03.domain.redacted}{VmBGMHhETPeovaMCKvB6YQ}{xPkABkOYR0OJ6-SaIw8PfQ}{os-data-03.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-data-02.domain.redacted}{iIwcH22cTC2Wvouj459mIw}{d6OuWakqQM-cfUePyCQBgA}{os-data-02.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-data-01.domain.redacted}{dJH-nBD1SsmOSmhr1UQ5BA}{cHE2Vex7R8SIg7KT7WPQTw}{os-data-01.domain.redacted}{ip-address}{d}{shard_indexing_pressure_enabled=true},{os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}{shard_indexing_pressure_enabled=true}}, term: 195, version: 383130, reason: ApplyCommitRequest{term=195, version=383130, sourceNode={os-master-01.domain.redacted}{bKV5xsTUSTa0iye-cdT5Zg}{glVIgSPVR_O32V8XU3_m1w}{os-master-01.domain.redacted}{ip-address}{m}} 2026-09-29T16:20:26.534+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [cluster.info.update.interval] from [30s] to [1m] 2026-09-29T16:20:26.534+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.low] from [95%] to [90%] 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,534][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.high] from [97%] to [95%] 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [plugins.index_state_management.metadata_migration.status] from [0] to [1] 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO][o.o.i.i.ManagedIndexCoordinator] [os-data-04.domain.redacted] Canceling metadata moving job because of cluster setting update. 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [plugins.index_state_management.template_migration.control] from [0] to [-1] 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.low] from [95%] to [90%] 2026-09-29T16:20:26.535+03:00 INFO [OpensearchProcessImpl] [2026-09-29T16:20:26,535][INFO][o.o.c.s.ClusterSettings] [os-data-04.domain.redacted] updating [cluster.routing.allocation.disk.watermark.high] from [97%] to [95%] ``` **3. What steps have you already taken to try and solve the problem?** Reread the documentation, relevant Opensearch documentation, looked for similar topics. **4. How can the community help?** Am I doing something wrong? Or is there some bug in data node configuration override? I know there is a warning on the Data Node Configuration Override documentation page that says the overrides are not fully supported by Graylog. Perhaps it’s not working as intended? ---
**Author:** ![ovreba](https://avatars.discourse-cdn.com/v4/letter/o/bc79bd/32.png) [@ovreba](https://community.graylog.org/u/ovreba)\ **Post date:** [October 5, 2026, 11:37am UTC](https://community.graylog.org/t/data-node-configuration-override-not-taking-effect/37617/2 "2026-10-05T11:37:01Z")
I was able to get around this by getting access to the API by generating a client certificate in the UI and adding it to admin\_dn as described in this post: > [@Give indices:admin/template/put permission to user datanode](https://community.graylog.org/t/give-indices-admin-template-put-permission-to-user-datanode/35519/6): > > Well, that was much much much easier than I initially thought. Really thanks Tomas Dvorak. openssl x509 -in cert\_mycert.crt -noout -subject subject=CN = datanode cat /etc/graylog/datanode/overrideconfig.conf plugins.security.authcz.admin\_dn = CN = datanode cat /etc/graylog/datanode/datanode.conf | grep overri opensearch\_configuration\_overrides\_file = /etc/graylog/datanode/overrideconfig.conf Then, restarted graylog-datanode and the CURL PUT command worked. I’ll be doing some tests now. … With this cert I am once again able to access the API and used it to set the watermarks via API: ```auto curl -sk --cert /etc/graylog/datanode/client-cert.crt --key /etc/graylog/datanode/client-cert.key -X PUT 'https://os-master-01.domain.redacted:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{ "persistent": { "cluster.routing.allocation.disk.watermark.low": "95%", "cluster.routing.allocation.disk.watermark.high": "97%", "cluster.routing.allocation.disk.watermark.flood_stage": "99%" } }' curl -sk --cert /etc/graylog/datanode/client-cert.crt --key /etc/graylog/datanode/client-cert.key 'https://os-master-01.domain.redacted:9200/_cluster/settings?pretty' { "persistent" : { "cluster" : { "routing" : { "rebalance" : { "enable" : "all" }, "allocation" : { "disk" : { "threshold_enabled" : "true", "watermark" : { "low" : "95%", "flood_stage" : "99%", "high" : "97%" } }, "enable" : "all" } }, "info" : { "update" : { "interval" : "1m" } } }, "opendistro" : { "index_state_management" : { "history" : { "number_of_replicas" : "1" } } }, "plugins" : { "index_state_management" : { "metadata_migration" : { "status" : "1" }, "template_migration" : { "control" : "-1" } } } }, "transient" : { } } ```