In our integration environment, there are two graylog nodes within a Kubernetes cluster. The process buffer is full (input buffer and output buffer are empty).
I have been trying to tune the performance by modifying the # of the processor for each buffer, besides increasing heap size. The # of out message did reach 3,000~4,000 per second for a while, then it dropped back to ~300 per second without any configuration change.
What may cause the process buffer full while others are empty?
Why the number of out messages changes while no configuration change?
What parameters should I try to config?