am working on processing a large amount of logs file with graylog and es. The input speed is about 100k/second， but the output speed to elastic search is just 7000/second. i have test the performance of my es cluster whose processing speed is about 14k/s, where es should not be bottleneck.
And I have checked the server load, the CPU usage is less than 5%, memory usage and network usage are both quite low.
I have tried the following potential configure changing:
- Change output buffer threads core pool size and max pool size to 30 and 90 respectively.
- Change output batch size to 2000
- Change processbuffer processors and outputbuffer processors to 128.
But unfortunately, there is no observably enhancement.
So what is wrong with my graylog and how to solve it?
64 Cores CPU
- Graylog Version: 2.2.0
- Elasticsearch Version: 2.4.3
- MongoDB Version: 3.*
- Operating System: Centos 6.7
- Browser version: Chrome