Tuning ES for consuming logs faster

Hi All ,

I’d like to know is there any specific setting need to be done to make ES behind the GL servers to consume incoming logs much faster. This is because i am facing buffer overflow issue on my fluentd setup on my central log server which sends to the graylog server. Upon starting the daemon of fluentd , within minutes it will start having buffer overflow issues. After some research, i understand that the ES needs to be tuned to ingest data faster than the regular setting , especially when you send 60gb of log per day. Below are the things i have tried and failed :frowning:

  1. single node setup with 20gb to 45gb on ES heap size & 16gb to 25gb heap size for GL
  2. 3 node setup for GL and 3 node setup for ES behind nginx with similar ram for heap as above
  3. 3 node GL and 4 node ES with above heap size
  4. setting inputbuffer_processors+processbuffer_processors+outputbuffer_processors = total cores in my server

Not sure what i am doing wrong here.Seeking your guidance on making solving this issue. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.