While I was standing up new app clusters recently there were some issues that caused many millions of log messages to be generated in short periods of time. At the time I though “those will age out fine”, but now that the FS usage alerts have begun I realize how badly I’ve stepped in it. The organic growth in our log volume will use all the available space before the messages age out. Add to that that the messages landed in a “catch-all” index that I can’t simply blow away.
The indexes in question are no longer live, and what I would like to do is run a filtered reindex operation to build a copy without the millions of trash entries. Conventional Elasticsearch wisdom seems to be “reindex old_42 to new_42, delete old_42, alias new_42 to old_42”, but I get the feeling that something is going to go wrong when Graylog wants to rotate out old_42 which isn’t an actual index anymore.
Will the alias work, or will I need to do a second reindex back to the original name?