1. Describe your incident:
ElasticsearchException[Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add  total shards, but this cluster currently has / maximum shards open;]]
2. Describe your environment:
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
trying to increase shards
4. How can the community help?
hi, i understand that i am out of shards, but i am not sure what to do about it. I was looking into increasing the shards, but i am now thinking i should change my indexes.
currently, i have about 12 indexes and all are set to rotate daily and retain for 90 days.
my first questions, is what does the community recommend for rotation strategies? should i be rotating less often? this would result in less shards…correct?
By default Graylog puts down four shards per index which gives it immediate compatibility with a clustered environment. If you aren’t’ clustered, you don’t need 4… The info in the post will help more, ask away if you need more clarification!
h, so i read everything over and i think i understand. I see that by default, an index is created with 4 shards. I am not quite clear on how i get to 1000 though, if i have 12 indexes x 4 shards/index…is that per rotation period?
in any event…how do i change this. If i edit an index and change it from 4 shards to 1…will the system do whatever is necessary to reduce the shards…do i have to do any cleanup after?
yup! so it’s 12 x (number of indices currently saved per index set) x 4
Editing the index set and reducing the shards will affect future indices on rotation, changing your index set rotation strategy to keep fewer indices will eliminate some you have currently. You COULD create an Elasticsearch cluster to take on the shards or for that matter change the number of clusters allowed… though the last one there is not recommended per Elasticsearch…
hi @gsmith , I think i saw that index sizes between 20 and 50 gig were usually ok. some of my indexes have very little activity, so i changed those to rotate weekly. the big ones, i left daily
as for replica’s, that is set to 0
btw, is there a way to empty (completely) an index in the GUI. some of my indexes i can afford to dump and free up shards…do i have to rm in the OS or can i use the GUI to clear it out?
You can delete each index manually in the GUI by viewing each indices you want to delete and clicking on the red “Delete Index” button. It is preferable to do it in the GUI so Graylog registers it but you can do it via elasticsearch if it is a large number (You would then “recalculate Index ranges”)