Second node no longer processing messages from journal

Have a two node cluster. Both were working but ran into an issue where mongo couldn’t start on the master node. It had filled the available disk so I moved the DB to another larger disk. Master node comes back up just fine and starts processing messages. Since then though the secondary node starts up with no problem and no errors in the log and even shows up in my “nodes” page of the graylog web interface. The problem now is that the main node shows it is ingesting and processing messages but the secondary one just shows that the journal contains 2+ million unprocessed messages in 1 segment and won’t start processing. Multiple restarts of both the service and the OS but still no progress. Any ideas? Anybody else have this problem or solved it? Here is a link to the image from the node page in the web interface.

HI Matt,

Can you please give a look at that node GL logs ? Anything interesting there? If you are not able to see anything in the logs, try increasing its verbosity.

Let me know

After finding nothing in the logs. Grasping at straws I just decided to clear the journal on that node and see what happened. Shut it down. Cleared journal and viola. Working again. Thank you for the response Scampuza.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.