Every hour IndexRangesCleanupPeriodical marks all but the current active index as unavailable. The indices are still there and if I recalculate the range for each index individually Graylog will recognize it until the next time IndexRangesCleanupPeriodical runs. I cannot recalculate all of the ranges at once, though. That option, under Maintenance, doesn’t have any effect.
I’ve checked the logs for Graylog, Elasticsearch, and MongoDB, but I don’t have any insight into what’s happening. Turning on debug mode for Graylog or Elasticsearch just flooded things with so many logs I was unable to find any useful needles in the haystack.
Has anyone experienced this or have any ideas on what to do, besides an hourly cronjob to recalculate ranges?
I’ve been able to downgrade our testing clusters to Elasticsearch 6.8.3, though they get unhappy if all of the nodes are not started at exactly the same time. I’ve tried various other versions of Elasticsearch 6.8.x and all of them, including 6.8.4 exhibit this same behavior.
I still can’t figure out why the version of Elasticsearch would cause Graylog to mark indices as unavailable. This is incredibly frustrating, because Graylog can’t search those indices, so every hour nearly all of our data disappears from the UI. The data is still there and Graylog can see that the indices exist, but it doesn’t have the time ranges, so the data is inaccessible.
