I had an ES failure during the weekend (ran out of space) which I resolved today.
Once I fixed ES graylog started ingesting the unprocessed logs from my other 30 graylog servers.
However /var was filled up and graylog stopped processing logs.
I then stopped graylog and moved the journal to a dedicated partion and rsynced all the files from /var/lib/graylog/journal to the new location.
I then started both nodes in the graylog cluster but now I have 5 million unprocessed messages.
I found the following post: After disk space issue - no out messages - help
I really don;t want to delete the journal because I’m assuming I will lose all the messages currently being stored within the journal. Is that assumption false?
I also noticed that the journal on node2 is not being populated like it is on node1.