So I have logstash and graylog set up in kubernetes (logstash pushes to graylog) and have an elasticsearch cluster for storage.
The problem I am facing is that, whenever my ES cluster goes red (due to storage problems), after fixing the ES cluster from red to green, graylog doesnt autorecover.
Graylog seems to be writing all logs perfectly, but no data is visible on the UI
Upon seeing the logs, this log is repeatedly seen.
After this I have to manually recalculate index ranges, after which suddenly all data for previous hours is visible. ( so it means it was being indexed correctly, but just the UI wasn’t able to show it).
I see it repeatedly tried to recover (by that deflector log), but isn’t able to for some reason. Needed help debugging this as I have some scripts running which make search queries on graylog and hence need the data there.
Can someone help me with this asap
Thanks & Regards