I had a storage issue on my virtual host where the host itself ran out of disk space.
After clearing the issues up and reclaiming the storage I rebooted both the Graylog Node and the Elasticsearch node (separate VMs).
Now in GL the journal has 1.4million messages queued and GL isn’t processing them. I assume the journal became corrupted (though I can’t see anything in the logs).
Looking through the forum at similar issues is the only solution to delete the journal directory on the GL server and therefore lose all the journaled messages?
@nick You can take the backup of journal directory other location and then delete the journal directory (default location: /var/lib/graylog-server/journal) on the Graylog server. After deleting you need to restart your Graylog server service to rotate the logs.
@nick You don’t need to do anything with the journal directory backup. Its just precautionary steps I told you to do. After deleting the journal directory you should restart Graylog server service and check if it started processing or not.
Is there anything I can do here or if you have a corrupted Journal you’ve lost the logs within that time? Seems odd that Graylog keeps adding to the Journal if its corrupted rather than stopping the ingest.
@nick Glad to know that your issue is resolved. Your journal is important until its went into Elasticsearch. Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough.
