Disk Full, Help with Cleanup

Graylog Central (peer support)
1

Problem Description

I’ve gotten myself into a mess. I have a separate partition to hold Elasticsearch data and I have a symbolic link to /var/lib/elasticsearch.
It’s now at 92% full. GL reports:

Elasticsearch nodes disk usage above low watermark
…For this reason Elasticsearch will not allocate new shards to the affected nodes. …

The system is now inoperable!

I have tried the following things:

  1. I was at 10 indices turning over ever 24 hours. I cut that down to 5. Still at 92% and inoperable.
  2. I chagned it to 40 indices turning over every 3 hours. Still at 92% and inoperable.
  3. I tried deleting some of the indices in /var/lib/elastic/search//var/lib/elasticsearch/nodes/0/indices. Still at 92% and inoperable.

What’s going on here? I’m literally deleting the indexes off the disk and I am still at 92% as reported by the df utility.

Thank you for your help.

Environmental information

Operating system information

centos-linux-release-8.3-1.2011.el8.noarch

Package versions

  • Graylog 4.0.7
  • MongoDB 4.2.14
  • Elasticsearch 7.10.2
2

OK. I figured it out.
Apparently the mount command doesn’t persist across reboots. I rebooted and the mount disappeared. GL started using the original /var/lib/elasticsearch directory, which was being held on my boot disk of only 200G. That was the disk that was full. My brand new 1000GB disk was partitioned, formatted, and sitting there unused.

I think that solves the problem. Now I have to figure out this fstab thing. :slight_smile:

3

Hello,
Depending on you OS here is an example of CentOS.
Once you use mount command go into your fstab file.

vi /etc/fstab

Add

/dev/sdb1 /mnt/path/folder ext4 defaults 0 0

Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   18M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root   83G  7.8G   75G  10% /
/dev/sda1                194M  126M   69M  65% /boot
/dev/sdb1                296G  236G   45G  85% /mnt/path/folder
tmpfs                    378M     0  378M   0% /run/user/1000

Save, Hope that helps

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.