After upgrading to version 2.3, I’ve had an issue for when Graylog applies the log retention plan (Set for 1D, keep 30 days), graylog does not reassign the shard to the newly created index. I’ve had to manually go reassign the shard to the correct index to get my ES cluster back to green, and then “Recalculate index range”.
My understanding is that all of Graylog’s configuration, while it’s running, is located in mongoDB, I’ve gone through every collection on my MongoDB, but there doesn’t seem to be anywhere that defines where the “Graylog_deflector” currently is supposed to be pointing to, other than in the browser. In the graylog logs the error that is being thrown is that it can’t point to the newest index because the ES is red, because whenever my log retention policy kicks in and creates a new index, it’s not automatically reassigning the shard when the new index is created, as you can see below. The AggregatesMaintenance is from a plugin that runs every minute looking for events with the same IP that have happened more times in x amount of minutes, could this be interrupting the shard allocation? Below is the log entry of when the errors begin.
017-09-20T19:00:06.084-05:00 INFO [AbstractRotationStrategy] Deflector index <Graylog> (index set <graylog_123>) should be rotated, Pointing deflector to new index now! 2017-09-20T19:00:06.085-05:00 INFO [MongoIndexSet] Cycling from <graylog_123> to <graylog_124>. 2017-09-20T19:00:06.085-05:00 INFO [MongoIndexSet] Creating target index <graylog_124>. 2017-09-20T19:00:06.138-05:00 INFO [Indices] Successfully created index template graylog-internal [MongoIndexSet] Waiting for allocation of index <graylog_124>. 2017-09-20T19:00:36.407-05:00 INFO [AggregatesMaintenance] removed 0 history items 2017-09-20T19:00:36.408-05:00 WARN [Aggregates] Indexer is not running, not checking any rules this run. 2017-09-20T19:01:06.240-05:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index