I need your advice here.
I have just realized that I’m running out of diskspace and thus have lowered the retention period down to 15 days (want to keep 2 weeks in total).
After that I have restarted both elasticsearch and graylog.
In the Elasticsearch log I see the following entry:
[2017-06-14 10:31:15,217][INFO ][cluster.routing.allocation.decider] [American Dream] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2017-06-14 10:31:45,223][WARN ][cluster.routing.allocation.decider] [American Dream] high disk watermark [90%] exceeded on [6N9daCofRaqkf_8iik3GJw][American Dream][/var/opt/graylog/data/elasticsearch/graylog/nodes/0] free: 4gb[8.1%], shards will be relocated away from this node
I still see the indices in Graylog (my hope was that they would get deleted during the restart).
Would I have to manually delete the indexes myself now to free up some diskspace?
For instance: I could easily delete index graylog_0 and graylog_1 as they contain data from 2 months ago that I no longer need. Shall I delete them manually through the Graylog UI?
what I don’t understand is that Elasticsearch is configured to delete old indices that it no longer needs to free up some diskspace.
My current configuration is:
Rotation strategy: Index Time
Rotation period: P1D
Retention strategy: Delete Index
Max number of indices: 8
However, my file system where the elasticsearch data is stored was 100% full this morning…is there something that I forgot to configure yet?
I’ve made my changes through the web-interface.
yep, I deleted them manually through the webui and managed to recover the elasticsearch cluster (it was in a red state previously due to the full filesystem). I will monitor the situation over the weekend
