Well this is weird. GL has been in our organisation for 18 months (kept up-to-date), and we’ve never had any problems of note.
Today however, I came in to the office after the weekend, and my cluster-state was red. I had a quick look at my indexes, and noticed that the current active index had not cycled. My max index size is 35GB, and this one had managed to rock up to 217GB.
Is there anywhere in particular I should be looking for the route cause of this? My ES cluster was at 90% capacity at the time (so around 400GB left on each of the three ES nodes).
I tried to manually rotate the active write index, but that just seemed to create more unassigned shards. I then deleted around 5 indexes from the bottom of the pile, and they started re-assigning. Weird, as to me it looked like there was plenty of space left on the ES cluster, but more importantly, what would cause it to not rotate in the first place? Is there some sort of limit imposed on the amount of ES storage that can be used possibly?