1. Describe your incident: My 3-node DataNode cluster experienced a disk-full event (1.3TB). Even after manually rotating and deleting indices via the Graylog UI, the disk space reported by df -h did not match du -sh.
Specifically, on Node 1, 2 and 3, df -h shows around 196GB used (16%) , but sudo du -h --max-depth=1 / only tallies 131GB . I tried service restart on Node 3, the df and du values tallied up (matched), but Node 1 and 2 continues to show a ~65GB gap.
2. Describe your environment:
OS Information:24.04
Package Version:Graylog DataNode (OpenSearch 2.19.3)
Storage: 1.3TB EBS Volumes
Cluster: 3 Nodes, GREEN status.
Hey @Arishem ,
In restarting the data-node service and running those commands again, can you see a difference?
Hello, @Wine_Merchant yes.
after sudo systemctl restart graylog-datanode
sudo du -sh /var/lib/graylog-datanode/
sudo du -B1 -s / 2>/dev/null | awk ‘{print “du total: " $1/1024/1024/1024 " GB”}’
df -h /
sudo ls -ld /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/*/ | wc -l
sudo du -sh /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/*/ | sort -rh | head -20
Does the below show Graylog hanging on to remnants of any deleted files/directories on the nodes with discrepancies?
sudo lsof +L1 | grep -i graylog*
show nothing in all datanode.
will it be the metadata eating the space?
i have 30 Indices,
boosty
January 20, 2026, 11:27am
9
Arishem
January 21, 2026, 2:09am
10
ow! @boosty thank you for sharing the link. i will look into it and thank you too! @Wine_Merchant
