# Users feedbacks / Guides for heavy load graylog Cluster **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [November 26, 2018, 10:12am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794 "2018-11-26T10:12:55Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 26, 2018, 10:12am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/1 "2018-11-26T10:12:55Z")
Hi everyones, i create this topic to have informations about sizing for a “heavy load” graylog cluster. Specially for the sizing of ressources (CPU / IO / RAM) of each nodes (graylog / elasticsearch), heap memory configuration …tricky options … I think it would be interesting for the community to have generals advises. * * * **Inside this Topic :** @macko003 (great performance / sizing review) : [Users feedbacks / Guides for heavy load graylog Cluster](https://community.graylog.org/t/users-feedbacks-for-heavy-load-graylog-cluster/7794/4?u=piellick) @benvanstaveren, review of his graylog cluster : [Users feedbacks / Guides for heavy load graylog Cluster](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/7?u=piellick) **Others Topics registry :** @jfowler : [Understanding how to use a Graylog Cluster](https://community.graylog.org/t/understanding-how-to-use-a-graylog-cluster/4385) @scampuza[Performance Tuning Whitepaper, Guide, Doc](https://community.graylog.org/t/performance-tuning-whitepaper-guide-doc/1835) @Aenima4six2 [Championing Graylog and need performance advice](https://community.graylog.org/t/championing-graylog-and-need-performance-advice/2309) @zahnd[Enhance Graylog search performance by adding new Elastic nodes?](https://community.graylog.org/t/enhance-graylog-search-performance-by-adding-new-elastic-nodes/2040) @arnaud [Best practice for index and shards configuration](https://community.graylog.org/t/best-practice-for-index-configuration/984) **External link:** Another (and great) external blog : [https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-graylog-lessons-learned/](https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-graylog-lessons-learned/) great Elasticsearch cluster design approach : [Designing the Perfect Elasticsearch Cluster: the (almost) Definitive Guide](https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-almost-definitive-guide-e614eabc1a87) loggly tips for Elastic : [https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/](https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/) from @jan, Elasticsearch shards sizing and considerations : > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/de/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... > **[Modern monitoring & analytics | Datadog](https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/)** > > See metrics from all of your apps, tools & services in one place with Datadog's cloud monitoring as a service solution. Try it for free. ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 26, 2018, 10:42am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/2 "2018-11-26T10:42:04Z")
> [@piellick](#): > > for a “heavy load” graylog cluster (near 100Go of logs per day). Paging @macko003, who apparently does 13TB each month 😃 ---
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 26, 2018, 10:45am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/3 "2018-11-26T10:45:18Z")
Gorgeous 😃 … @macko003 free to provide any feedbacks to the community. ---
**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\ **Post date:** [November 26, 2018, 12:30pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/4 "2018-11-26T12:30:40Z")
Based on my performance monitoring, it’s not “heavy load”, but maybe useful //I mentioned these, but here is at one place. Traffic: 4-6k log/sec ~ 350-400.000.000 log/day peaks when a missconfigured device sends all its log - 1-2.000.000/min - no problem ~40-45 MB/s load balancer output if traffic to GL servers **EDIT2:** We have 40+ streams, 600+ sources, 50+ different source type First of all, We planned it geo redundant, so calculate with it. At the moment our system: 2 load balancer - nginx (we will change it to ipvs) - 2 vcpu/2GB mem 4 graylog servers - 8 vcpu, 16GB mem - 10GB heap EDIT3: @jan suggest somewhere not more than 2 GB heap, BUT We checked the HEAP with 4GB, but the java’s GC run more often and longer, and under the GC the OS UDP error counters increase more than the 10GB HEAP. 10 Elasticsearch servers - 8 vcpu, 32 GB mem, 16 GB Heap - 3TB storage ~10 MB/s daily max write speed. 40MB/s not problem 3 mongodb - 2 vcpu, 2 GB mem In GL: ~50 input ~40 streams, ~ 10 with syslog output 50+ extractors 10 pipelines My experiences loadbalancer, mongo servers do nothing 🙂 , no load on it Graylog can process immediately all messages (except peaks) (0% in/out/process buffer usage on every node), 15% night, 20% daytime cpu usage. Although you get better performance if you increase the output\_batch\_size in GL, don’t do it. [https://github.com/Graylog2/graylog2-server/issues/5091](https://github.com/Graylog2/graylog2-server/issues/5091) I think at the moment the elastic is the battleneck in our system. 20% CPU usage on every node. We can increase the cpu and the mem next. You need about 2-3% ES heap of the stored data without replica to have a usable search. In out system one single word search takes about 5 sec with the histogram drawing in the 35 days data monitor all performance data what you can (I think it could be another good topic) I did some stress tests on a cloned test system about 1 year ago, with loggen, the 50k log/sec won’t be problem with 4 ES servers I hope slowly the administrators will start to use the system, and increase the amount of logs, and I can start to do some performance optimization My “special” settings: GL: //live without elastic connection for a weekend message\_journal\_max\_age = 48h message\_journal\_max\_size = 95gb ES: //for geo redundant data storage node.attr.site: A cluster.routing.allocation.awareness.force.site.values: A,B cluster.routing.allocation.awareness.attributes: site //the GL bug over http.max\_content\_length: 250mb Backup: Every day elastic snapshot for every IS and index to NFS Some evidence by photoshop 🙂 (At the first, you see the data with replica, under IS without replica, other IS names not public) ![graylog-indices](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/ca79a6db23e65c600034d1d76343bb788becff12.png) (daily) ![graylog-daily](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/bab69ee2bf4a330f465abe19b8b6846453360c11.png) **EDIT :** Colleagues start to log a bit more. At the moment 13-15k log/sec ~x3 of the previous daytime traffic GL: 6%-\> 20% CPU; ES output average process time 200-\>230k us; /node ES: 6%-\>20%; 4-\>11 MBps disk IO; 300-\>900 ms index time; 0-\>4 ms fetch time /node LB: 10%-\>20% CPU; 47-\> 130 mbps interface bandwidth (total) As I see, we will need to increase the ES disk size. ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 26, 2018, 12:41pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/5 "2018-11-26T12:41:23Z")
Odd… clicking that little ❤ icon more than once just makes it go _ba-bump,ba-bump_… 😃 I can only upvote you once @macko003 ---
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 26, 2018, 12:47pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/6 "2018-11-26T12:47:54Z")
So many useful informations 😍 thanks @macko003 ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\ **Post date:** [November 29, 2018, 9:47am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/7 "2018-11-29T09:47:55Z")
Not entirely sure if our setup classifies as “heavy load” but here goes. Keep in mind we run everything on bare metal (yes, cloud is great, but sometimes it isn’t 😃 ) Currently running as follows: 3 Graylog servers (24 core cpu, 128Gb memory, 32Gb heap allocated to Graylog which doesn’t seem to be an issue; also 12 processbuffer\_processors and 4 outputbuffer\_processors) 25 Elasticsearch servers (19 data, 3 master, 3 routing). Data nodes are 64Gb memory, quad-core CPU, 32Gb heap for ES, 2x4Tb raid 0 storage for data. Master and routing nodes are 32Gb memory, quad core CPU, 16Gb heap for ES. 1 Mongodb instance on each Graylog server in a replica set. Graylog itself runs with 2 inputs, currently 50-odd streams, with about 30 single-stage pipelines, and 5 pipelines with multiple stages and a heck of a lot of grokking/lookup tabling going on. The reason for the number of data servers is two-fold: first, we have hiliarious retention policies on some of this data (60 days is the norm), and we run most index sets with 3 replicas because the data is important - as well as the boost to search speed it brings. We’re currently pushing a consistent 2500-3000 msg/sec through. Graylog’s journal is set to a max age of 48h, and 512Gb since that is about the average size of logs we pull in across that time so at least I can be gone for a weekend or something and heroically save the world on a Monday. Backup wise we don’t actually back up the ES data on account of the replicas we run (accepted risk), index archival/closure/retention is handled by our in-house snapshot manager that uses the Graylog API to find indices that contain “stale” data, and snapshots them to S3 before removing them. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 29, 2018, 10:16am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/8 "2018-11-29T10:16:03Z")
Thank you for this topic. I want to add this two links: Sharding and size calculation in Elasticsearch > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/de/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... Fix unassigned shards in Elasticsearch > **[Modern monitoring & analytics | Datadog](https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/)** > > See metrics from all of your apps, tools & services in one place with Datadog's cloud monitoring as a service solution. Try it for free. ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\ **Post date:** [November 29, 2018, 11:12am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/9 "2018-11-29T11:12:38Z")
Since for legacy reasons we have daily indices, we run at 4 shards per index given our average volume of 100Gb daily (100/30 = 3.33 = 4 shards of 30Gb a piece should cover it) - we may switch to using size-based indices given that Graylog manages it all (in our previous setup we used an external tool to manage index closure/archiving). ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 29, 2018, 11:16am UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/10 "2018-11-29T11:16:47Z")
> [@benvanstaveren](#): > > we have hiliarious retention policies on some of this data `*cough*` Try five frackin’ years… That’s how long our security-related logs need to be retained ☹ Technically it’s easily doable, but it sure is gonna get messy. It’s tempting to make an output that just barfs those into a logfile. ---
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 29, 2018, 12:12pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/11 "2018-11-29T12:12:18Z")
Thanks Jan. i update the main post. ---
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 29, 2018, 12:20pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/12 "2018-11-29T12:20:30Z")
Interesting, and your long-term retention logs fils for security purposes are saves on tapes ? ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\ **Post date:** [November 29, 2018, 12:23pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/13 "2018-11-29T12:23:07Z")
We also keep 5 years but after 6 months we shuffle everything off to a nice cozy S3 bucket… bit more manageable that way 😃 ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 29, 2018, 12:33pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/14 "2018-11-29T12:33:07Z")
Pffft, heck no 😃 Tapes? We don’t have those… With luck I’ll shift them into a dead file on a filesystem. Ditto for S3 and other cloud storage: our env is 100% off-the-grid, so we’re stuck with what we have available. ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\ **Post date:** [November 29, 2018, 12:35pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/15 "2018-11-29T12:35:36Z")
That sounds so… James Bond-ish, somehow. ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 29, 2018, 1:07pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/16 "2018-11-29T13:07:51Z")
![](https://cdck-file-uploads-us1.s3.dualstack.us-west-2.amazonaws.com/flex016/uploads/graylog/original/2X/8/869de6aa1848e7b139004e960e43f8405e68773c.jpeg) 😛 ---
**Author:** ![piellick](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@piellick](https://community.graylog.org/u/piellick)\ **Post date:** [November 29, 2018, 1:36pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/17 "2018-11-29T13:36:00Z")
mouhahaha tape is cheap and not so disgusting for very long retention 😛 ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\ **Post date:** [November 29, 2018, 1:38pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/18 "2018-11-29T13:38:11Z")
Dat ga je een mede-Nederlander toch niet aandoen… \>.\> But yeah off-the-grid entirely seems like you need yourself a heck of a lot of storage boxes to keep up with demand 😃 ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [November 29, 2018, 6:28pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/19 "2018-11-29T18:28:14Z")
> [@benvanstaveren](#): > > Dat ga je een mede-Nederlander toch niet aandoen… \>.\> _Adoeh! Je weet dat Belanda onderling soms best pittig kunnen reageren, toch?_ 😉 But let’s get back on-topic 🙂 Our environment is far from heavy-load: I’m expecting less than 10GB a day. Our difficulty lies in both the long retention times as well as our network topology, which is both highly dispersed and has an expectancy of network loss. I still need to look into queueing servers that can be used to locally cache the logging in case the Graylog receivers become unreachable. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [November 29, 2018, 6:38pm UTC](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794/20 "2018-11-29T18:38:54Z")
use beats - that can handle if the receiver is not available. Or if bound to syslog something like written here: > **[Graylog2/graylog-guide-syslog-amqp](https://github.com/Graylog2/graylog-guide-syslog-amqp)** > > How to use send Syslog messages via AMQP to Graylog - Graylog2/graylog-guide-syslog-amqp [Next page](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794.md?page=2)