Overloaded Backlog

So. I made a big mistake while importing some IIS logs. And now there’s a huge backlog of message processing.

It has been like this for 24 hours now. And the unprocessed messages count has actually gone up overnight (from 60-odd million to 80+).

I stopped the filebeat that was sending in the flood of messages as soon as I realized what I did yesterday.

Is there a way to kill this and return to normal operation?

I couldn’t find a way to gracefully get out of this situation. The backlog was approaching 100M messages.

I did find a few posts about just clearing the journal. That worked. The downside is that anything in that journal is lost.

  • Go to /var/lib/graylog-server.
  • Stop the graylog service.
  • Rename (or delete it if you’re feeling lucky) the journal directory.
  • Start the Graylog service.

Graylog should return to normal operation. But there will be a gap in the history.

image