# Graylog Cluster, Buffer process 100% stop process messages **URL:** https://community.graylog.org/t/graylog-cluster-buffer-process-100-stop-process-messages/1032 **Category:** Graylog Central (peer support) **Created:** [May 3, 2017, 5:07pm UTC](https://community.graylog.org/t/graylog-cluster-buffer-process-100-stop-process-messages/1032 "2017-05-03T17:07:03Z") **Posts on this page:** 1 **Showing post:** 1
### Author: ![rafaelcarsetimo](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rafaelcarsetimo/32/215_2.png) [@rafaelcarsetimo](https://community.graylog.org/u/rafaelcarsetimo) #### Post date: [May 3, 2017, 5:07pm UTC](https://community.graylog.org/t/graylog-cluster-buffer-process-100-stop-process-messages/1032/1 "2017-05-03T17:07:03Z")
Hello Sirs, I know there are several reports of this case, but I am following all the possibilities that I found in the forum, but without success. Basically, I have 3 nodes. They have a processing capacity each at about 1600m / s. But intermittently one of them stop process the messages but still send to journal… and the only way to re-process is to restart the Graylog service. But after doing this, a few minutes later, another node stops processing the messages and again I have to restart the Graylog service. The problem occurs in all nodes after a non periodic time, and one by one. I believed that it happened because of a bad message formation of a Fortigate / Fortinet log, but I corrected it with help here from the Forum, treating the messages as RAW. No logs from graylog and elasticsearch, even in debug mode give me clues to what’s going on. **3 Nodes with: 16 VCPUS, 24GB RAM, FC Disc Storage 3PAR - CentOS 7 Updated** ## CONFS **conf\_graylog\_node\_1** is\_master = true node\_id\_file = /etc/graylog/server/node-id password\_secret = xxxxxxxxxxxxxxx root\_username = admin root\_password\_sha2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx root\_timezone = America/Sao\_Paulo plugin\_dir = /usr/share/graylog-server/plugin rest\_listen\_uri = [http://192.168.0.195:9000/api/](http://192.168.0.195:9000/api/) web\_listen\_uri = [http://192.168.0.195:9000/](http://192.168.0.195:9000/) rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 20000000 elasticsearch\_max\_number\_of\_indices = 20 retention\_strategy = delete elasticsearch\_shards = 4 elasticsearch\_replicas = 1 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_cluster\_name = graylog elasticsearch\_discovery\_zen\_ping\_unicast\_hosts = 192.168.0.195:9300, 192.168.0.196:9300, 192.168.1.187 elasticsearch\_cluster\_discovery\_timeout = 15000 elasticsearch\_network\_host = 192.168.0.195 elasticsearch\_discovery\_initial\_state\_timeout = 10s elasticsearch\_analyzer = standard output\_batch\_size = 500 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 16 outputbuffer\_processors = 3 processor\_wait\_strategy = blocking ring\_size = 65536 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 2 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir = /var/lib/graylog-server/journal lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb://graylog1:27017,graylog2:27017,graylog3:27017/graylog mongodb\_max\_connections = 1000 mongodb\_threads\_allowed\_to\_block\_multiplier = 5 content\_packs\_dir = /usr/share/graylog-server/contentpacks content\_packs\_auto\_load = grok-patterns.json proxied\_requests\_thread\_pool\_size = 32 **conf\_graylog\_node\_2** is\_master = false node\_id\_file = /etc/graylog/server/node-id password\_secret = xxxxxxxxxxxxxxx root\_username = admin root\_password\_sha2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx root\_timezone = America/Sao\_Paulo plugin\_dir = /usr/share/graylog-server/plugin rest\_listen\_uri = [http://192.168.0.196:9000/api/](http://192.168.0.196:9000/api/) web\_listen\_uri = [http://192.168.0.196:9000/](http://192.168.0.196:9000/) rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 20000000 elasticsearch\_max\_number\_of\_indices = 20 retention\_strategy = delete elasticsearch\_shards = 4 elasticsearch\_replicas = 1 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_cluster\_name = graylog elasticsearch\_discovery\_zen\_ping\_unicast\_hosts = 192.168.0.195:9300, 192.168.0.196:9300, 192.168.1.187 elasticsearch\_cluster\_discovery\_timeout = 15000 elasticsearch\_network\_host = 192.168.0.196 elasticsearch\_discovery\_initial\_state\_timeout = 10s elasticsearch\_analyzer = standard output\_batch\_size = 500 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 16 outputbuffer\_processors = 3 processor\_wait\_strategy = blocking ring\_size = 65536 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 2 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir = /var/lib/graylog-server/journal lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb://graylog1:27017,graylog2:27017,graylog3:27017/graylog mongodb\_max\_connections = 1000 mongodb\_threads\_allowed\_to\_block\_multiplier = 5 content\_packs\_dir = /usr/share/graylog-server/contentpacks content\_packs\_auto\_load = grok-patterns.json proxied\_requests\_thread\_pool\_size = 32 **conf\_graylog\_node\_3** is\_master = false node\_id\_file = /etc/graylog/server/node-id password\_secret = xxxxxxxxxxxxxxx root\_username = admin root\_password\_sha2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx root\_timezone = America/Sao\_Paulo plugin\_dir = /usr/share/graylog-server/plugin rest\_listen\_uri = [http://192.168.1.187:9000/api/](http://192.168.1.187:9000/api/) web\_listen\_uri = [http://192.168.1.187:9000/](http://192.168.1.187:9000/) rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 20000000 elasticsearch\_max\_number\_of\_indices = 20 retention\_strategy = delete elasticsearch\_shards = 4 elasticsearch\_replicas = 1 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_cluster\_name = graylog elasticsearch\_discovery\_zen\_ping\_unicast\_hosts = 192.168.0.195:9300, 192.168.0.196:9300, 192.168.1.187 elasticsearch\_cluster\_discovery\_timeout = 15000 elasticsearch\_network\_host = 192.168.1.187 elasticsearch\_discovery\_initial\_state\_timeout = 10s elasticsearch\_analyzer = standard output\_batch\_size = 500 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 16 outputbuffer\_processors = 3 processor\_wait\_strategy = blocking ring\_size = 65536 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 2 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir = /var/lib/graylog-server/journal lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb://graylog1:27017,graylog2:27017,graylog3:27017/graylog mongodb\_max\_connections = 1000 mongodb\_threads\_allowed\_to\_block\_multiplier = 5 content\_packs\_dir = /usr/share/graylog-server/contentpacks content\_packs\_auto\_load = grok-patterns.json proxied\_requests\_thread\_pool\_size = 32 **elasticsearch\_conf\_node\_1** cluster.name: graylog node.name: graylog1.example.com.br network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: [“192.168.0.195”, “192.168.0.196”, “192.168.1.187”] index.refresh\_interval: 30s index.translog.flush\_threshold\_ops: 50000 **elasticsearch\_conf\_node\_2** cluster.name: graylog node.name: graylog2.example.com.br network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: [“192.168.0.195”, “192.168.0.196”, “192.168.1.187”] index.refresh\_interval: 30s index.translog.flush\_threshold\_ops: 50000 **elasticsearch\_conf\_node\_3** cluster.name: graylog node.name: graylog3.example.com.br network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: [“192.168.0.195”, “192.168.0.196”, “192.168.1.187”] index.refresh\_interval: 30s index.translog.flush\_threshold\_ops: 50000 **etc\_sysconfig\_elasticsearch ALL NODES** ES\_HEAP\_SIZE=12g ES\_STARTUP\_SLEEP\_TIME=5 **etc\_sysconfig\_graylog-server ALL NODES** JAVA=/usr/bin/java GRAYLOG\_SERVER\_JAVA\_OPTS="-Xms6g -Xmx6g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow" GRAYLOG\_SERVER\_ARGS="" GRAYLOG\_COMMAND\_WRAPPER="" **Thread dump of node 558b0d80 / graylog2.example.com.br** NOT PROCESSING NOW [https://paste.ee/p/AU1ff](https://paste.ee/p/AU1ff) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/2281b738b5aa81b2c78ff62d4e86b55194134435.png) --- _[View the full topic](https://community.graylog.org/t/graylog-cluster-buffer-process-100-stop-process-messages/1032)._