Journal Filling but Graylog not Processing

I had a storage issue on my virtual host where the host itself ran out of disk space.

After clearing the issues up and reclaiming the storage I rebooted both the Graylog Node and the Elasticsearch node (separate VMs).

Now in GL the journal has 1.4million messages queued and GL isn’t processing them. I assume the journal became corrupted (though I can’t see anything in the logs).

Looking through the forum at similar issues is the only solution to delete the journal directory on the GL server and therefore lose all the journaled messages?

@nick You can take the backup of journal directory other location and then delete the journal directory (default location: /var/lib/graylog-server/journal) on the Graylog server. After deleting you need to restart your Graylog server service to rotate the logs.

Hope this information helps you :slight_smile:

Sorry, what do I do with the backup of the journal directory?

Do I backup the Journal directory, restart the service and then restore the backup? Or do I then need to restart the service again after the restore?

@nick You don’t need to do anything with the journal directory backup. Its just precautionary steps I told you to do. After deleting the journal directory you should restart Graylog server service and check if it started processing or not.

Yes, all is working again - how do I now import the logs from the backup of the journal I took?

I don’t want to lose 1.6 million logs from the last 24 hours

Is there anything I can do here or if you have a corrupted Journal you’ve lost the logs within that time? Seems odd that Graylog keeps adding to the Journal if its corrupted rather than stopping the ingest.

@nick Glad to know that your issue is resolved. Your journal is important until its went into Elasticsearch. Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough.

Hi, sorry but that doesn’t make any sense?

I know why the journal was filling, I’m not concerned about that.

Can I somehow take the Journal files I have backed up and import the logs they contain into Graylog or are these logs now lost?