I currently run 5 Graylog nodes on cluster on aws c5 ec2 instances. 16 CPU and 32 GB Ram.
On these machines I also run elastic coordinate only nodes.
Heap size for Graylog is 12G
Heap size for ES coordinate node 8G
http.max_content_length of elastic set to 1024Mb
Graylog nodes configured to send messages to all 5 coordinate only nodes.
Coordinate only nodes are part of elastic cluster that consist of 16 data nodes and 3 separate masters.
Data nodes have 3.5 Tb NVME SSD, 16 cores and 122 GB RAM.
I set output batch size to 1000
Refresh rate: 1s
Max elastic connections to 160 and
Max connections per route 32.
Our median log flow is 15.000msg/sec
Does it make sense to raise batch size to 10.000 or it may have negative effect on performance due to very large bulk size ?