Retention strategy change

Graylog
#1

Hey all, question on retention changes. At this point I’m not sure if what im seeing is the correct behaviour or not.

GL Version 4.0.6+40b7be5, codename Noir

So take this index set:
Rotation: time
Period: P1D
Srategy: close
Max indeces: 180

That is working all fine and good but now I made a change to it to the following settings:
Rotation: time
Period: P1D
Srategy: delete
Max indeces: 274

The problem is that the original strategy ran for long enought that it racked up 368 indeces (181 of which are now open, 187 of which are closed). Now we a higher max than before for opened indeces, but it has more indeces total than its max. Should it automatically delete the closed indeces past 274 or, becuse those are closed, they are not being looked at and would need to be manually deleted?

#2

Hello,

Have you tried to Recalculate index ranges or Rotate active write index?

#3

@gsmith Manually I have not but as I look at that index zet this morning it did auto recalculate and rotate 13 hours ago.

What prompted me to make this change was the following message

Elasticsearch nodes disk usage above low watermark
There are Elasticsearch nodes in the cluster running out of disk space, their disk usage is above the low watermark. For this reason Elasticsearch will not allocate new shards to the affected nodes

Im leaning towards there being multiple factors to preventing retention strategy from activating: space, strategy change, number of indeces vs max set size. I think i will start manually deleting the older indeces to get to a point where size goes below the treshhold and see how it behaves after that

#4

@sgarciam

If you manually delete Indices make sure you do it through the Web UI.

#5

@gsmith
Found yet another factor that was preventing things fromn running as they should. Storage running low coincided quite nicely with the max number of shards allowed going over so it was complaining about both. I’ve upped the max numbers of shards allowed (not ideal but needed to get things rotated) and everything rotated as it should after a graylog service restart.

Well… too late for that. did it through curl calls to the elasticsearch node. Not exeedingly worried about that (unless I should be). However, I do have a second index set with exactly the same details as stated in the original post: Lots of closed indeces, strategy changed to delete, closed indeces still there above the increased max indeces. Havent touched that one just yet besides it finally rotating by itself after increasing the number of shards allowed mentioned above.

What I would really like to know is if the retention strategy considers closed indeces for its calculation. If it doesnt it would explain it not removing the old indeces but i dont know its exact behaviour in this situation.

If I have to manually delete indeces, are ther curl calls to the graylog server that I could use to do that? I would really nike not to go through the web ui to delete 200+ indeces.

#6

Hello,

Well, yes there is, but it would probably take you just as long
Lists Indices:

curl -XGET http://localhost:9200/_cat/shards

Delete Indices

curl -XDELETE 'http://localhost:9200/name_of_index/'

Graylog Server Service might need to be restarted.

It probably doesn’t consider close indices, so you may need to open them up and recalculate the indexes again. I personally haven’t done that, but you could give it a try.