Hello Graylog community,
today I want to share a problem I already solved.
The problem was that graylog pauses indexing for no apparent reason when one of the ES nodes is down for maintenance and index.unassigned.node_left.delayed_timeout is still in effect.
It looks like this: Messages are comming in, but no message is going out.
All my logging indices have replicas, so from my understanding indexing should work.
There are no replicas for unused indices like graylog_1, but that should not matter, right?
The cause was simple after I looked at /var/log/graylog-server/server.log
There was one index I call “trash” that had replicas disabled.
This index set does not receive many messages, but becase, I think, graylog’s message journal works first-in-first-out it took just about 100 messages destined for this index set to completely block indexing.
So that’s that. Solved by adding a replica for trash.
