I have been running a decent size Graylog instance 25,000 msg/s for the last few months without incident. As of last night we started to experience the process buffers filling up. Eventually it appears that the deflector dies and stops processing messages all together. The elasticsearch cluster is green and doesn’t appear to be having performance issues. It doesn’t appear that the messages are even getting to the output buffer. Messages are as a result stacking up in the journal.
I tried default settings and the following to help with the process buffers filling up.
processbuffer_processors = 8
output_batch_size = 100
ring_size = 262144
Any guidance how how I can dig in to see what is causing the process buffers to fill up would be helpful. The logs are not pointing me anywhere currently.