Please see my below cluster setup
Server Type 1 - 16 core, 32 GB Memory
Server Type 2 - 16 core, 64GB Memory
Server Type 1 - 3 Nodes
Graylog, Kafka, Elasticsearch - master nodes , 4gb heap
Server Type 2 - 2 Nodes
Elasticsearch - data nodes, 30gb heap
Graylog reads from kafka in server type 1.
ES is using less than 50% of the allocated heap.
I have about 20 input configured on Graylog and most of them are type Gelf kafka.
I have about 300 alerts configured on the 20 streams.
Graylog processing the messages is very slow. I get output of only of max 2500 messages per second and becomes a bottleneck. I can see the number of unprocessed messages going very high and graylog output is very slow.
How can I improve the graylog message processing output?