am working on processing a large amount of logs file with graylog and es. The input speed is about 100k/second， but the output speed to elastic search is just 7000/second. i have test the performance of my es cluster whose processing speed is about 14k/s, where es should not be bottleneck.
And I have checked the server load, the CPU usage is less than 5%, memory usage and network usage are both quite low.
I have tried the following potential configure changing:
Change output buffer threads core pool size and max pool size to 30 and 90 respectively.
Change output batch size to 2000
Change processbuffer processors and outputbuffer processors to 128.
But unfortunately, there is no observably enhancement.
So what is wrong with my graylog and how to solve it?
you run on a very old system, I suggest to update it. The ES has 2 new main version with a lot of improvements at the speed side also. As Jan mentioned
1+1. nodes information (also mentioned by jan)
Elasticsearch doesn’t handle 384 GB of RAM. Max 64GB for ES.
you didn’t share any config. but the eg the ES heap size is important.
your config changes -
4.2 - GL has a bug if you send bigger batch then your ES http request size, it can stops the process. Take care about it.
4.3 - please chekc the server.conf’s comments. DON1T use more buffer processors then your cpu’s number.
If you don’t have experience with it, I suggest use the defaults.
If you have speed problems on the ES side it won’t help.
