Performance Tuning Whitepaper, Guide, Doc


(Scampuza) #1

Hi there!

I have a production cluster of GrayLog, moving an average of 2 million messages per day. It has been working like a charm, but sometimes, when a message peak arrives, the servers behave weird, they start to hold messages in the Journal, and they take a lot of time to flush that journal.

Is there any white paper, guide or documentation for tuning the servers? server.conf file has a lot of tuning parameters, but most of them are like blackboxes for me.

I’ve Googled a lot trying to find docs about tuning , but GrayLog is kind of new.

I appreciate any help with this.


(Bashere1) #2

I faced the same issue that you are when initially scaling Graylog. There seems to be a lack of published documentation on large scale installations and tuning. We are pushing about 25,000 to 40,000 msg/s though our Graylog currently. The settings I found that helped the most are as follows.

ring_size = 262144 (must fit in your L3 cache and must be a power of 2)

If your cpu cores are not being utilized you can try to increase the following.

processbuffer_processors = {see what works}
outputbuffer_processors = {see what works}

I also set my heap size to 6G out of a total of 10G on the box.

If your output buffers are filling up you are going to want to look at expanding or tuning your elastic cluster.


(Scampuza) #3

Thanks for your quick response !

Do you have ES installed on a separate cluster?

Thanks in advance


(Scampuza) #4

Hi and thanks !

Can you please share with me your specifications for the GL and ES cluster? Number of nodes, CPU, memory, JVM settings, etc ?

I would appreciate if you can share with me this info

Thanks


(Bashere1) #5

Yes, Graylog and Elastic run on different servers. Graylog is the bottleneck for us currently as we are running quite a few of inputs / extractors that are rather CPU intensive.

Graylog is 8CPU/10GB RAM @ 9 VM’s
Elastic is 8CPU/32GB RAM @ 12 VM’s


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.