I’ve recently set up a production Graylog server after running a PoC test with the pre-built OVA. I have devices pointing to it already, receiving somewhere between 1000-2000 messages per second on average.
Everything works fine until I add an output to our All Messages stream, to forward the logs to our SIEM. The output performance will drop from processing messages as received to maybe 10-50 messages per second. If I remove the output and reboot the graylog service it will jump up to process upwards of 20,000 messages per second to clear everything that was backed up. If I disable our loudest Input, reducing the log intake to 10-200 messages per second, the same issue occurs and it will drop to processing 5-20 messages per second. At no point are system resources heavily utilized. The OVA did not have this issue when testing the All Messages stream output.
I’m not sure what needs to be reviewed or where to figure out why adding the output reduces the processing performance on this server. Any help would be appreciated.