Hi All ,
I’d like to know is there any specific setting need to be done to make ES behind the GL servers to consume incoming logs much faster. This is because i am facing buffer overflow issue on my fluentd setup on my central log server which sends to the graylog server. Upon starting the daemon of fluentd , within minutes it will start having buffer overflow issues. After some research, i understand that the ES needs to be tuned to ingest data faster than the regular setting , especially when you send 60gb of log per day. Below are the things i have tried and failed
- single node setup with 20gb to 45gb on ES heap size & 16gb to 25gb heap size for GL
- 3 node setup for GL and 3 node setup for ES behind nginx with similar ram for heap as above
- 3 node GL and 4 node ES with above heap size
- setting inputbuffer_processors+processbuffer_processors+outputbuffer_processors = total cores in my server
Not sure what i am doing wrong here.Seeking your guidance on making solving this issue. Thanks