September 28, 2021, 9:04pm
I’ve gotten myself into a mess. I have a separate partition to hold Elasticsearch data and I have a symbolic link to /var/lib/elasticsearch.
It’s now at 92% full. GL reports:
Elasticsearch nodes disk usage above low watermark
…For this reason Elasticsearch will not allocate new shards to the affected nodes. …
The system is now inoperable!
I have tried the following things:
I was at 10 indices turning over ever 24 hours. I cut that down to 5. Still at 92% and inoperable.
I chagned it to 40 indices turning over every 3 hours. Still at 92% and inoperable.
I tried deleting some of the indices in /var/lib/elastic/search//var/lib/elasticsearch/nodes/0/indices. Still at 92% and inoperable.
What’s going on here? I’m literally deleting the indexes off the disk and I am still at 92% as reported by the df utility.
Thank you for your help.
Operating system information
September 28, 2021, 9:45pm
OK. I figured it out.
Apparently the mount command doesn’t persist across reboots. I rebooted and the mount disappeared. GL started using the original /var/lib/elasticsearch directory, which was being held on my boot disk of only 200G. That was the disk that was full. My brand new 1000GB disk was partitioned, formatted, and sitting there unused.
I think that solves the problem. Now I have to figure out this fstab thing.
September 28, 2021, 11:26pm
Depending on you OS here is an example of CentOS.
Once you use mount command go into your fstab file.
/dev/sdb1 /mnt/path/folder ext4 defaults 0 0
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 18M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 83G 7.8G 75G 10% /
/dev/sda1 194M 126M 69M 65% /boot
/dev/sdb1 296G 236G 45G 85% /mnt/path/folder
tmpfs 378M 0 378M 0% /run/user/1000
Save, Hope that helps
October 12, 2021, 11:27pm
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.