Hey all, question on retention changes. At this point I’m not sure if what im seeing is the correct behaviour or not.
GL Version 4.0.6+40b7be5, codename Noir
So take this index set:
Rotation: time
Period: P1D
Srategy: close
Max indeces: 180
That is working all fine and good but now I made a change to it to the following settings:
Rotation: time
Period: P1D
Srategy: delete
Max indeces: 274
The problem is that the original strategy ran for long enought that it racked up 368 indeces (181 of which are now open, 187 of which are closed). Now we a higher max than before for opened indeces, but it has more indeces total than its max. Should it automatically delete the closed indeces past 274 or, becuse those are closed, they are not being looked at and would need to be manually deleted?
@gsmith Manually I have not but as I look at that index zet this morning it did auto recalculate and rotate 13 hours ago.
What prompted me to make this change was the following message
Elasticsearch nodes disk usage above low watermark There are Elasticsearch nodes in the cluster running out of disk space, their disk usage is above the low watermark. For this reason Elasticsearch will not allocate new shards to the affected nodes
Im leaning towards there being multiple factors to preventing retention strategy from activating: space, strategy change, number of indeces vs max set size. I think i will start manually deleting the older indeces to get to a point where size goes below the treshhold and see how it behaves after that
@gsmith
Found yet another factor that was preventing things fromn running as they should. Storage running low coincided quite nicely with the max number of shards allowed going over so it was complaining about both. I’ve upped the max numbers of shards allowed (not ideal but needed to get things rotated) and everything rotated as it should after a graylog service restart.
Well… too late for that. did it through curl calls to the elasticsearch node. Not exeedingly worried about that (unless I should be). However, I do have a second index set with exactly the same details as stated in the original post: Lots of closed indeces, strategy changed to delete, closed indeces still there above the increased max indeces. Havent touched that one just yet besides it finally rotating by itself after increasing the number of shards allowed mentioned above.
What I would really like to know is if the retention strategy considers closed indeces for its calculation. If it doesnt it would explain it not removing the old indeces but i dont know its exact behaviour in this situation.
If I have to manually delete indeces, are ther curl calls to the graylog server that I could use to do that? I would really nike not to go through the web ui to delete 200+ indeces.
Graylog Server Service might need to be restarted.
It probably doesn’t consider close indices, so you may need to open them up and recalculate the indexes again. I personally haven’t done that, but you could give it a try.
Ah ok. Was afraid I had done it some other way that was not a good idea but that is how I did it. took no time at all. And yes the Graylo Server Service had to be restarted.
