# Datanode remove: what exactly happens? **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** data-node\ **Created:** [May 8, 2026, 5:12pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249 "2026-05-08T17:12:16Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 8, 2026, 5:12pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/1 "2026-05-08T17:12:16Z")
\*\*1. Describe your incident: \*\* My setup: 1 graylog headnode + 1 datanode. shards: 1 replicas: 0. I’m trying to add another datanode and retire the old one. Is it possible to do with graylog web GUI without going into API stuff? If I add another datanode and remove the old one, would I loose data? Or else graylog simply makes sure all the data from old datanode is migrated to new one before actually removing it? **2. Describe your environment:** - OS Information: Linux - Package Version: 6.1.8 - Service logs, configurations, and environment variables: shards:1 replicas: 0 **3. What steps have you already taken to try and solve the problem?** I searched the forum, and googled. Documentation is minimal on datanode and remove action. **4. How can the community help?** **Helpful Posting Tips:** [Tips for Posting Questions that Get Answers](https://community.graylog.org/t/tips-for-posting-questions-that-get-answers/21828) [Hold down CTRL and link on link to open tips documents in a separate tab] ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [May 11, 2026, 9:11am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/2 "2026-05-11T09:11:59Z")
This operation will require you to interface with the Opensearch API, you can do this be pulling a client cert from the Graylog UI under cluster config. ![Screenshot 2026-05-11 at 10.11.00](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/e/2/e2c79a2c3e02a6cb66de764c7d5f4e34ad57fac7.png) Assuming the second Graylog Data Node has been added to the cluster, you could disable routing to the older node with the below. ```auto PUT _cluster/settings { "transient": { "cluster.routing.allocation.exclude._ip": "OLD_DATA_NODE_IP" } } ``` Once that is done you should be able to see the shards drain off of the old node down to zero. `GET _cat/shards?v` Assuming the indices are comprised of a single shards and without replicas this should be a safe operation. Once the node is drained of shards it can be shutdown. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 11, 2026, 6:30pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/3 "2026-05-11T18:30:53Z")
Last time I tried, API thing didn’t work. I’ll generate a new 3rd party cert and try again. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 11, 2026, 6:32pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/4 "2026-05-11T18:32:36Z")
@Wine_Merchant what’s is your guess in regards to removing a datanode: Does graylog do it gracefully or else in my case, it would lead to a data loss? I guess my question is about what goes on under the hood, when I click ‘Remove’. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 12, 2026, 3:31am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/5 "2026-05-12T03:31:09Z")
Managed to move all shards to new datanode via API by using a new client cert with all\_access role. Thanks @Wine_Merchant . I’m still not able to retire the old node though. There are two complications: - turns out Graylog has some internal shards that it uses which’re set to non-zero replicas. In any case, these replicas can be discarded since they have primaries. - Before removing old datanode, I can’t make sure graylog is in a good state since if the node is disabled, graylog throws ‘cluster\_manager\_not\_discovered\_exception’. it looks like graylog is working ok but the new datanode can’t take over the datanode cluster master role due to quorum issue I think. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 12, 2026, 5:06am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/6 "2026-05-12T05:06:21Z")
I"ve create a new issue for this: > [@Shrink 2-node graylog cluster to 1 (graylog@6.1)](https://community.graylog.org/t/shrink-2-node-graylog-cluster-to-1-graylog-6-1/37271): > > \*\*1. Describe your incident:\*\* Trying to go from 2 datanodes to 1. Let’s label datanodes as dn1 & dn2. all shards are located to dn2 via an API call. When I click “Stop” for dn1 on web UI, graylog UI seems to work ok but throws this error: cluster\_manager\_not\_discovered\_exception . current manager is dn1, it seems whatever I do, dn2 isn’t able to assume the cluster manager role. 2. Describe your environment: OS Information: Linux Package Version: 6.1 Service logs, configurations, and… ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [May 12, 2026, 8:07am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/7 "2026-05-12T08:07:03Z")
Hey @Sinan, Glad you are one step further and nice work on creating another thread. I’ve commented there. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 13, 2026, 7:02pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/8 "2026-05-13T19:02:20Z")
Now that I think about this again, shouldn’t Graylog web UI automatically handle routing when I click “Remove” and if it’s not possible prompt an error to the user. ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [May 14, 2026, 10:29am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/9 "2026-05-14T10:29:17Z")
The issue is removing a node from a two node cluster will always be an issue as quorum requires at least two voting nodes to elect a leader. ![Screenshot 2026-05-14 at 11.27.32](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/f/d/fd0bb3a642e0e6f498fbde0c922c892a914259d5.png) ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 14, 2026, 5:28pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/10 "2026-05-14T17:28:23Z")
makes sense. So if quorum is not issue, i.e. if I have 3 datanodes, graylog will automatically handle allocation of shards if I click remove on a datanode? ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [May 15, 2026, 8:23am UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/11 "2026-05-15T08:23:40Z")
Hey @Sinan I ran a quick test, spun up a 3 node cluster and created an index with 3 shards which were equally placed across all three nodes. Upon removal of node1, cluster remains green and node1’s shards are now on node3. It appears that adding the node1 back to the cluster, shards require a manual rebalance to move them back. ---
**Author:** ![Sinan](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@Sinan](https://community.graylog.org/u/Sinan)\ **Post date:** [May 15, 2026, 6:06pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/12 "2026-05-15T18:06:13Z")
good to know. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [May 29, 2026, 6:06pm UTC](https://community.graylog.org/t/datanode-remove-what-exactly-happens/37249/13 "2026-05-29T18:06:50Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.