Graylog Deflector not repointing itself

[sahilcool]

So I have logstash and graylog set up in kubernetes (logstash pushes to graylog) and have an elasticsearch cluster for storage.
The problem I am facing is that, whenever my ES cluster goes red (due to storage problems), after fixing the ES cluster from red to green, graylog doesnt autorecover.
Graylog seems to be writing all logs perfectly, but no data is visible on the UI
Upon seeing the logs, this log is repeatedly seen.

After this I have to manually recalculate index ranges, after which suddenly all data for previous hours is visible. ( so it means it was being indexed correctly, but just the UI wasn’t able to show it).
I see it repeatedly tried to recover (by that deflector log), but isn’t able to for some reason. Needed help debugging this as I have some scripts running which make search queries on graylog and hence need the data there.

Can someone help me with this asap

Thanks & Regards

[sahilcool]

Any idea related to this?

[gsmith]

Hey @sahilcool

@sahilcool

manually recalculate index ranges

try manually rotating you index set called “graylog_37120”

[sahilcool]

Yes, it works by manually recalculating index ranges/rotating index set, but my question was why is there a need to manually intervene, why isn’t it able to automatically recover?

[gsmith]

Hey @sahilcool

I could be a couple reseaon why it does that. When Opensearch/elasticsearch goes RED from the lack of space, it goes into Read mode. I would dig through logs MongoDb, Graylog and OS/ES find anything that would pertain to that issue. Perhaps look at connection between Graylog & OS/ES cluster… Im not sure how you have everything setup to troubleshoot your issue.