Deleting Old Indexes (not seen by Graylog) from the File System

[mdobson90 (Matt Dobson)]

Hi,

Having managed to get Graylog working since my previous post (Unable to Add or Delete Enterprise License - Error 400) AND upgraded to the latest version, I was having a look through and found something odd.

I received an alert that my Graylog Server was down to only 10% space left on HDD.
I have a 1.2TB HDD for the server (ubuntu 16.04 VM), which is showing ~1000GB used.

Looking at the graylog server web ui, I see:

This indicates that Graylog is only using 213Gb for storing indexes. Even with the OS and installed software etc, this should mean the total system is well under 300Gb.

Doing a bit of digging led me to find the /var/lib/elasticsearch/nodes/0/indices folder.
Running du -sh shows this folder is 810Gb - the culprit.

a) why are there indices in here which aren’t being picked up by Graylog (roughly 550-600GB).
b) can I manually delete any of these?

If the answer to b) is “yes”, how do i work out which index is linked to which folder within nodes/0/indices?

I can always add more space to the server, but as these logs arent being detected by Graylog, I’d rather remove them (or re-index them) to either make backups smaller, or see the data in Graylog.

Matt

[jan (Jan Doberstein)]

you should interact with the elasticsearch API

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/cat-indices.html

This way you know what is in what directory plus you can delete that indices via API and you will not destroy your installation.

[mdobson90 (Matt Dobson)]

Not destroying my installation is definitely a positive.
I’ll have a dig.

Happy Xmas to the Graylog Team btw!!

[macko003]

You just close the indices instead of deleting.
GL will count only the opened indices.

[mdobson90 (Matt Dobson)]

Thanks - Turns out I hadn’t clicked on the actual “Default Index Set” name/button, rather just the “edit / actions” menu, they were visible all along.

Using Jan’s suggestion of ElasticsearchAPI, I’d already seen which ones were closed/open, but when I went into the Default Index Set, I was able to see all the open/closed indices and delete as required.