Deleted Indices do not free HD Space

Graylog Central (peer support)
1

1. Describe your incident:

Having new Graylog 7 Single Node with Datanode on Ubuntu 24
Index Rotation is Legacy 10x4.7GB, 1 Shard, 0 Replica, Delete (not Close)
Automatically deleted Indeces do not free the Space, also not after some hrs.
There are noc closed indexec left

2. Describe your environment:

  • OS Information:

  • Package Version: graylog-7.0-repository_latest.deb

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
compared with df and du command
Restart of datanode makes space free

**4. How can the community help?

  • any other solved this?**
  • known problem for GL7/datanode?
2

Hi @Gerald

Could be a bug, but are the files still seen with $ lsof | grep deleted ?

lsof +L1 could show if a program is attached to the deleted file.

3

Hi Arie,

i’ve collected some infos:
around 1.1GB in Journal, nothing in index
Delta between du and df currently 63GB

root@:/# lsof | grep deleted | wc -l 

54214 

root@:/# lsof | grep 'graylog-server/journal/messagejournal-0' | grep deleted | wc -l 

54060 

root@:/# lsof | awk '/graylog-server\/journal\/messagejournal-0/ && /deleted/ && $7 ~ /^[0-9]+$/ { 

    count++; 

    sum += $7; 

} 

END { 

    print "Anzahl Dateien:", count; 

    print "Gesamt (Bytes):", sum; 

    print "Gesamt (MiB):",  sum / 1024 / 1024; 

}' 

Anzahl Dateien: 170 

Gesamt (Bytes): 1206156530 

Gesamt (MiB): 1150.28 

root@:/# lsof | awk ' 

/graylog-server\/journal\/messagejournal-0/ && /deleted/ && $7 ~ /^[0-9]+$/ { 

    size = $7; 

    file = $NF; 

  

    if (file ~ /\.log\.deleted$/) { 

        log_count++; 

        log_sum += size; 

    } else if (file ~ /\.index\.deleted$/) { 

        idx_count++; 

        idx_sum += size; 

    } 

} 

END { 

    print "LOG-Dateien:   ", log_count, "Stück,", log_sum/1024/1024, "MiB"; 

    print "INDEX-Dateien: ", idx_count, "Stück,", idx_sum/1024/1024, "MiB"; 

}' 

LOG-Dateien:    1 Stück, 99.999 MiB 

INDEX-Dateien:  170 Stück, 1056.55 MiB 

 

root@:/# du -xh --max-depth=2 /data 

138G    /data/graylog/graylog-datanode 

479M    /data/graylog/mongodb 

161M    /data/graylog/graylog-server 

18M     /data/graylog/log 

138G    /data/graylog 

16K     /data/lost+found 

138G    /data 

root@:/# df -hT /data 

Filesystem                    Type  Size  Used Avail Use% Mounted on 

/dev/mapper/data--vg-data--lv ext4  295G  201G   79G  72% /data 

 

 

Lsof-Output Example: 

java      179874 342803 OkHttp             graylog  176u      REG              252,0  104856671    6553685 /data/graylog/graylog-server/journal/messagejournal-0/00000000002803546135.log.deleted 

java      179874 343595 OkHttp             graylog  DEL       REG              252,0               6555128 /data/graylog/graylog-server/journal/messagejournal-0/00000000002790700765.index.deleted 

java      179874 343595 OkHttp             graylog  DEL       REG              252,0               6555132 /data/graylog/graylog-server/journal/messagejournal-0/00000000002790619434.index.deleted 

java      179874 343595 OkHttp             graylog  mem       REG              252,0     166456    6554573 /data/graylog/graylog-server/journal/messagejournal-0/00000000002803546135.index.deleted
4

Hi @Gerald

Sorry for the late response, it looks like that the application does not let go of the file, so the actual deletion does not take place. What could be done is restart the accommodating services, and see if disk-space is is feed, second is a reboot and see if disk-space is freed.

If this happens, or not, it can point in the direction of ownership or permissions for the file(s) by inheritance from above folders, or a problem wit graylog / datanode what you could report as a possible bug at the github channel of graylog.

5

Hi Arie,
yes by restarting graylog-datanode the space is released.
After there’s no file entry (also not deleted files) with this amount can be find it looks like a bug.
I’ve also tried to disable mmap. no change.

I’ll do a report to Graylog.

Thanks and Best Regards Gerald

6

Yes, same here, after upgrading to datanode 7.0

I noticed that deletion happens when I stop datanode, not in the restart.

7

Hello,

I’m also having problems since upgrading from Graylog 6.3.5 to 7.0.1 using Datanode. Since the upgrade, disk space on the host is not being released, except when I stop or restart only the Datanode. In my case Graylog is running in Docker, but I’m seeing the same issue that was mentioned. Also I upgraded to 7.0.2 and the problem still exists.

I’ve also checked with lsof | grep deleted, but there is no output (no files marked as “deleted”). It seems like this might be a bug.

8

I have added my finding here: