We had new scenario pop up with our cluster over the weekend.
Sunday our ES master nodes became unreachable, this normally wouldn’t cause any issues, they would journal then clear out when things returned to a normal state.
In this instance the index also attempted to rotate while the ES masters were unavailable. It appears that incoming messages did not get journaled. We have some logging around this so we are alerted when the journal gets utilized and it remained at zero utilization throughout the incident.
The journal location is empty, no additional disk usage was logged by our hardware monitoring and I am not finding anything in graylog’s logs that indicate anything about journaling events either.
Can you suggest some things to look at?