Having managed to get Graylog working since my previous post (Unable to Add or Delete Enterprise License - Error 400) AND upgraded to the latest version, I was having a look through and found something odd.
I received an alert that my Graylog Server was down to only 10% space left on HDD.
I have a 1.2TB HDD for the server (ubuntu 16.04 VM), which is showing ~1000GB used.
Looking at the graylog server web ui, I see:
This indicates that Graylog is only using 213Gb for storing indexes. Even with the OS and installed software etc, this should mean the total system is well under 300Gb.
Doing a bit of digging led me to find the /var/lib/elasticsearch/nodes/0/indices folder.
Running du -sh shows this folder is 810Gb - the culprit.
a) why are there indices in here which aren’t being picked up by Graylog (roughly 550-600GB).
b) can I manually delete any of these?
If the answer to b) is “yes”, how do i work out which index is linked to which folder within nodes/0/indices?
I can always add more space to the server, but as these logs arent being detected by Graylog, I’d rather remove them (or re-index them) to either make backups smaller, or see the data in Graylog.