Elasticsearch Retention Period


(theresa) #1

Hi guys,

I need your advice here.
I have just realized that I’m running out of diskspace and thus have lowered the retention period down to 15 days (want to keep 2 weeks in total).
After that I have restarted both elasticsearch and graylog.

In the Elasticsearch log I see the following entry:
[2017-06-14 10:31:15,217][INFO ][cluster.routing.allocation.decider] [American Dream] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2017-06-14 10:31:45,223][WARN ][cluster.routing.allocation.decider] [American Dream] high disk watermark [90%] exceeded on [6N9daCofRaqkf_8iik3GJw][American Dream][/var/opt/graylog/data/elasticsearch/graylog/nodes/0] free: 4gb[8.1%], shards will be relocated away from this node

I still see the indices in Graylog (my hope was that they would get deleted during the restart).
Would I have to manually delete the indexes myself now to free up some diskspace?

For instance: I could easily delete index graylog_0 and graylog_1 as they contain data from 2 months ago that I no longer need. Shall I delete them manually through the Graylog UI?


(Jan Doberstein) #2

Hej Theresa,

what is your rotation and retention strategy you have configured?

In your Graylog Server you can always delete Indices yourself.


(theresa) #3

Hej Jan,

what I don’t understand is that Elasticsearch is configured to delete old indices that it no longer needs to free up some diskspace.
My current configuration is:

Rotation strategy: Index Time
Rotation period: P1D
Retention strategy: Delete Index
Max number of indices: 8

However, my file system where the elasticsearch data is stored was 100% full this morning…is there something that I forgot to configure yet?
I’ve made my changes through the web-interface.


(Jan Doberstein) #4

Hej @micsnare

I do not know why this happens - or better not happens. Did you try to delete the indices manually
?

Jan


(theresa) #5

Hej @jan

yep, I deleted them manually through the webui and managed to recover the elasticsearch cluster (it was in a red state previously due to the full filesystem). I will monitor the situation over the weekend :wink:

theresa.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.