Dear all, which are the best settings in case of 2000 messages/sec? heap, elasticsearch and mongodb settings? is there a tuning guideli es to increase performance?
Hey @marcob
This is a broad question. really depend on the environement,resources, Log type, hardware, etc…
Probably your best bet is look through the forum and see what other have done like this.
I personall would start witha Dev setup and start adjust the settings. So when this is in production you would know what to do.
Hey @marcob,
we have a case of 4000msg/s here and the magic eight 
The GL-Heap size is 8GB, Elastic-Heap 8GB, Disk-Journal ist max 8GB. Storage is, not 8GB but 6TB.
And all this on a single system with a SAN Storage Disk. The disk had to be moved to an older SAN (10k SAS-Disks) for maintenance, and it was noticeable. Otherwise it runs on full-flash Storage.
I hope it helps you?
Thanks, what about elasticsearch tuning? Or Graylog process optimizations? We have 4vCPU with 16GB RAM , HEAP 4GB. With all the INPUT enabled no more 400/500 msg/sec
for example:
processbuffer_processors = 5
outputbuffer_processors = 3
Are ok for a 4vCPU Virtual Machine?
my environment:
VMM = Proxmox/KVM (2x Xeon CPU E5-2630 v4 @ 2.20GHz, 2x 10Gbit NIC, 2x 8Gbit FC)
vCPU = 8
vRAM = 32GByte
processbuffer_processors = 6
outputbuffer_processors = 4
Active Rules = 33 (6 Pipelines)
OS = Debian 11 x86_64
GL Version = 4.3
Load average ~ 5.0
With a heap size of 4GB and 4 vCPU, 2000msg should be possible (also depends on the set of pipeline rules and regex) and look the comment of @gsmith
Is there a problem with processing 2000msg/s? → can you take a screenshot of the system/nodes/details path?
Thanks for asking @marcob, You’ve received several excellent response, so mine is based purely on the documentation. Perhaps it can help if you need to look up more information on your topic.
According to the Graylog documentation, tuning the heap size, Elasticsearch settings, and MongoDB settings can help increase performance when dealing with high message throughput. For example, it recommends increasing the heap size to at least 12GB when dealing with a message rate of 2000 messages/sec.
Regarding Elasticsearch, it’s recommended to increase the index and search threads and also increase the number of shards and replicas. For MongoDB the recommendations include increasing the wiredTiger cache size and adjusting the write concern.
Additionally, the documentation also provides a guide to “Performance Tuning” where you can find more information on how to optimize Graylog for high message throughput. This guide includes best practices for configuring Graylog, Elasticsearch, and MongoDB, as well as tips for monitoring and troubleshooting performance issues.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.