We noticed we’re running out of space so we proceeded to delete two of the five (now set to 4) indices.
One was deleted with curl, the other through the Graylog GUI. When checking the space with df -m, /dev/dm-0 has the same size and is growing.
A reboot of the server was performed, but we’re still running out of space (fast).
What additional steps need to be taken?
@CypherBit
I also had that happen in my environment. To find my problem on what was taking up all the storage space I execute the following command;
root# du --max-depth=5 /* | sort -rn | more du --max-depth=5 /* | sort -rn | more
I found out it was in my /var/log , I had like a 700Gb log file from Graylog Server. The origin was from an Input on the server kicking out messages.
@gsmith thank you for your reply.
It appears something else is taking up the space, log folder is quite small and the deleted indices no longer seem to be present, but running df -m shows that /dev/dm-0 is almost full:
df -m
Filesystem 1M-blocks Used Available Use% Mounted on
udev 1495 1 1495 1% /dev
tmpfs 300 1 300 1% /run
/dev/dm-0 15282 13223 1261 92% /
none 1 0 1 0% /sys/fs/cgroup
none 5 0 5 0% /run/lock
none 1500 0 1500 0% /run/shm
none 100 0 100 0% /run/user
/dev/sda1 236 121 103 55% /boot
/dev/sdb1 100664 57858 37670 61% /var/opt/graylog/data
du -a | sort -n -r | head -n 10
72745801 .
69358404 ./var
59246188 ./var/opt
59246184 ./var/opt/graylog
59183616 ./var/opt/graylog/data
58604564 ./var/opt/graylog/data/elasticsearch
58604560 ./var/opt/graylog/data/elasticsearch/graylog
58604556 ./var/opt/graylog/data/elasticsearch/graylog/nodes
58604552 ./var/opt/graylog/data/elasticsearch/graylog/nodes/0
58604536 ./var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices
curl http://localhost:9200/_cat/indices
yellow open graylog_4 V_dJkHHQR1SNXCFEXrYoCQ 4 1 13407604 388 16gb 16gb
yellow open graylog_3 uP3WTcSJSJ6ntEev7kcMOw 4 1 20000019 0 18.4gb 18.4gb
yellow open graylog_2 dylgfaS1Qk6SVbi7pe8hgA 4 1 20000017 0 20.4gb 20.4gb
How exactly did you delete these indices?
What’s the complete output of the command @gsmith provided in his post?
du --max-depth=5 /* | sort -rn
The first one graylog_0 was deleted using curl. graylog_1 was deleted through the GUI using the Delete index button.
This is the output (1st page):
du --max-depth=5 /* | sort -rn | more du --max-depth=5 /* | sort -rn | more
du: cannot access ‘/proc/943/task/1471/fd/96’: No such file or directory
du: cannot access ‘/proc/12640/task/12640/fd/4’: No such file or directory
du: cannot access ‘/proc/12640/task/12640/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/12640/fd/4’: No such file or directory
du: cannot access ‘/proc/12640/fdinfo/4’: No such file or directory
du: No such file or directory
--max-depth=5: No such file or directory
69682136 /var
59470336 /var/opt
59470332 /var/opt/graylog
59407764 /var/opt/graylog/data
58820460 /var/opt/graylog/data/elasticsearch
58820456 /var/opt/graylog/data/elasticsearch/graylog
9398472 /var/log
9382500 /var/log/graylog
8979460 /var/log/graylog/elasticsearch
1434016 /usr
981008 /opt
981004 /opt/graylog
813764 /lib
691932 /usr/lib
656396 /lib/modules
599688 /opt/graylog/embedded
554876 /var/cache
477632 /usr/src
459828 /var/opt/graylog/data/mongodb
451116 /usr/lib/jvm
371916 /opt/graylog/embedded/jre
371248 /usr/lib/jvm/java-8-oracle
319592 /var/log/graylog/server
277912 /var/cache/apt
250920 /var/opt/graylog/data/mongodb/journal
249536 /var/lib
224240 /opt/graylog/mongodb
224120 /opt/graylog/mongodb/bin
205480 /lib/modules/4.2.0-41-generic
205416 /lib/modules/4.2.0-35-generic
205036 /lib/modules/4.2.0-27-generic
203524 /usr/share
202408 /var/opt/graylog/data/mongodb/diagnostic.data
201440 /opt/graylog/embedded/jre/jre
201396 /usr/lib/jvm/java-8-oracle/jre
200916 /lib/modules/4.2.0-41-generic/kernel
200856 /lib/modules/4.2.0-35-generic/kernel
200748 /opt/graylog/embedded/jre/jre/lib
200700 /usr/lib/jvm/java-8-oracle/jre/lib
200476 /lib/modules/4.2.0-27-generic/kernel
185880 /var/lib/apt
185820 /var/lib/apt/lists
185692 /var/cache/apt/archives
181200 /var/cache/oracle-jdk8-installer
152856 /opt/graylog/embedded/lib
148508 /lib/modules/4.2.0-41-generic/kernel/drivers
148476 /lib/modules/4.2.0-35-generic/kernel/drivers
148380 /lib/modules/4.2.0-27-generic/kernel/drivers
140672 /usr/lib/x86_64-linux-gnu
135976 /opt/graylog/embedded/jre/lib
135824 /usr/lib/jvm/java-8-oracle/lib
128000 /lib/firmware
125040 /var/opt/graylog/data/etcd
125036 /var/opt/graylog/data/etcd/member
121569 /boot
107824 /usr/src/linux-headers-4.4.0-53
103468 /usr/src/linux-headers-4.2.0-35
103464 /usr/src/linux-headers-4.2.0-41
103456 /usr/src/linux-headers-4.2.0-27
98968 /opt/graylog/embedded/lib/ruby
90356 /var/cache/apt-xapian-index
90352 /var/cache/apt-xapian-index/index.2
There’s over 9 GB worth of data in the /var/log/graylog directory. 
True, but one of the indices has/had around 20GB, since I deleted two, I should have gained almost 40GB.
Can I just delete the contents of the /var/log/graylog directory?
The indices are on a different disk partition on your machine.
Thank you, I don’t know what I was looking at.
After checking the /var/log/graylog/elasticsearch folder there are tons of .log files, can these just be deleted?
If you don’t require the log files of Elasticsearch anymore, you can delete them.
If you’re unsure, try compressing them first (using xz or bzip2).
Thank you both so much, problem solved!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.