# HELP! Is Graylog datanode acting normal? **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** data-node\ **Created:** [January 16, 2026, 7:42am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862 "2026-01-16T07:42:45Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 16, 2026, 7:42am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/1 "2026-01-16T07:42:45Z")
**1. Describe your incident: Disk space discrepancy and “Ghost Files” on Graylog DataNodes.** My 3-node DataNode cluster experienced a disk-full event (1.3TB). Even after manually rotating and deleting indices via the Graylog UI, the disk space reported by `df -h` did not match `du -sh`. Specifically, on Node 1, 2 and 3, `df -h` shows around **196GB used (16%)**, but `sudo du -h --max-depth=1 /` only tallies **131GB**. I tried service restart on Node 3, the `df` and `du` values tallied up (matched), but Node 1 and 2 continues to show a ~65GB gap. ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/a/7/a75a925e5a55c25e488d29437ccc58ace471c69d.png) ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/1/1/11bb073b10752e1135bf73a27997057e1e8765a0.png) **2. Describe your environment:** - OS Information:24.04 - Package Version:Graylog DataNode (OpenSearch 2.19.3) - **Storage:** 1.3TB EBS Volumes - **Cluster:** 3 Nodes, GREEN status. ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [January 16, 2026, 2:13pm UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/2 "2026-01-16T14:13:36Z")
Hey @Arishem, In restarting the data-node service and running those commands again, can you see a difference? ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 18, 2026, 8:09am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/3 "2026-01-18T08:09:12Z")
Hello, @Wine_Merchant yes. example. df -h shows 196G used du -h shows 131G used after sudo systemctl restart graylog-datanode df -h shows 131G. du -h shows 131G. ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 18, 2026, 8:24am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/4 "2026-01-18T08:24:31Z")
sudo du -sh /var/lib/graylog-datanode/ 563G /var/lib/graylog-datanode/ sudo du -B1 -s / 2\>/dev/null | awk ‘{print “du total: " $1/1024/1024/1024 " GB”}’ du total: 567.866 GB df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 1.3T 633G 626G 51% / sudo ls -ld /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/\*/ | wc -l 264 sudo du -sh /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/\*/ | sort -rh | head -20 16G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/yRTuRVYaTNeG-jmcsE9bwQ/ 16G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/NhJsMFMQQPurzw\_xtdBB1Q/ 14G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/szc9mYzdQReXRJ4mrKl0zw/ 14G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/SVivBRIoSIWo3ygChEuCmg/ 13G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/QnSyLc5-S1el82rPWmURbg/ 13G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/Q-LUCfbCTqmeP5IsjkBOOg/ 13G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/H3x5KqsuTsK8XcGjDFzckg/ 13G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/7xkhOTwFTNO1LkhqqBx6Mg/ 12G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/YgGllgxvQ1CyFVTMGpFSGg/ 11G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/w2SnM0WEQziVGgj4RlZdtw/ 11G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/v2BhqgGXS1mXeLaEcf-GoQ/ 11G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/j3FaK1RrTEeh3pmmB9waqQ/ 11G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/1qk8pb4MQGawISdlnN1ESw/ 9.6G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/YO27Y-4yTb6hpkEoMxON7Q/ 9.5G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/xqcYYP9ZReiP-GLafKzK-g/ 9.5G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/3n3UxJgvQDeyWmmFBWMBgg/ 9.3G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/7YwSkqLRSniCcFTOR7WhAQ/ 8.7G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/BFXaNWxcQC2JMMDxuJpqdA/ 8.2G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/b5IRIhmdR9awkrxZMafwpw/ 8.2G /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/MSAE5\_AZQA2nsrMJxS9Cwg/ ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [January 19, 2026, 11:42am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/5 "2026-01-19T11:42:48Z")
Does the below show Graylog hanging on to remnants of any deleted files/directories on the nodes with discrepancies? `sudo lsof +L1 | grep -i graylog*` ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 20, 2026, 4:14am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/6 "2026-01-20T04:14:05Z")
show nothing in all datanode. ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 20, 2026, 4:14am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/7 "2026-01-20T04:14:46Z")
will it be the metadata eating the space? ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 20, 2026, 4:18am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/8 "2026-01-20T04:18:57Z")
i have 30 Indices, each indices Rotation period 6 hours and Max number of indices 12 ---
**Author:** ![boosty](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/boosty/32/17585_2.png) [@boosty](https://community.graylog.org/u/boosty)\ **Post date:** [January 20, 2026, 11:27am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/9 "2026-01-20T11:27:13Z")
Related issue in the OpenSearch repo: [[BUG] OpenSearch 2.19.3 (possibly other versions) not freeing disk space as expected until service is exited/terminated · Issue #20244 · opensearch-project/OpenSearch · GitHub](https://github.com/opensearch-project/OpenSearch/issues/20244) ---
**Author:** ![Arishem](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Arishem](https://community.graylog.org/u/Arishem)\ **Post date:** [January 21, 2026, 2:09am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/10 "2026-01-21T02:09:32Z")
ow! @boosty thank you for sharing the link. i will look into it and thank you too! @Wine_Merchant ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [February 4, 2026, 2:09am UTC](https://community.graylog.org/t/help-is-graylog-datanode-acting-normal/36862/11 "2026-02-04T02:09:52Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.