# Unprocessed messages **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [December 21, 2017, 8:56am UTC](https://community.graylog.org/t/unprocessed-messages/3588 "2017-12-21T08:56:36Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![n07n0w](https://avatars.discourse-cdn.com/v4/letter/n/f17d59/32.png) [@n07n0w](https://community.graylog.org/u/n07n0w)\ **Post date:** [December 21, 2017, 8:56am UTC](https://community.graylog.org/t/unprocessed-messages/3588/1 "2017-12-21T08:56:36Z")
Hello, We have: 1 graylog server, 4 elasticsearch nodes (1 balancer and 3 data), and ~20.000 msg/second. Everything works fine, but process buffer is full all the time, and ~3.000.000 unprocessed messages { “cluster\_name” : “gl2”, “status” : “green”, “timed\_out” : false, “number\_of\_nodes” : 4, “number\_of\_data\_nodes” : 3, “active\_primary\_shards” : 80, “active\_shards” : 80, “relocating\_shards” : 0, “initializing\_shards” : 0, “unassigned\_shards” : 0, “delayed\_unassigned\_shards” : 0, “number\_of\_pending\_tasks” : 0, “number\_of\_in\_flight\_fetch” : 0, “task\_max\_waiting\_in\_queue\_millis” : 0, “active\_shards\_percent\_as\_number” : 100.0 } is\_master = true node\_id\_file = /etc/graylog/server/node-id plugin\_dir = /usr/share/graylog-server/plugin rest\_listen\_uri = rest\_transport\_uri = web\_enable = true web\_listen\_uri = elasticsearch\_hosts = elasticsearch\_connect\_timeout = 2s elasticsearch\_socket\_timeout = 60s elasticsearch\_max\_total\_connections = 2000 elasticsearch\_max\_total\_connections\_per\_route = 2000 elasticsearch\_max\_retries = 2 rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 50000000 elasticsearch\_max\_number\_of\_indices = 5 retention\_strategy = delete elasticsearch\_shards = 24 elasticsearch\_replicas = 1 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_analyzer = standard output\_batch\_size = 10000 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 16 outputbuffer\_processors = 8 processor\_wait\_strategy = blocking ring\_size = 262144 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 2 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir =nal lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb: mongodb\_max\_connections = 1000 mongodb\_threads\_allowed\_to\_block\_multiplier = 5 content\_packs\_dir = content\_packs\_auto\_load = grok-patterns.json proxied\_requests\_thread\_pool\_size = 32 Is it normal, and if not how to fix it? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [December 21, 2017, 9:47am UTC](https://community.graylog.org/t/unprocessed-messages/3588/2 "2017-12-21T09:47:40Z")
you are missing the following Information: - Graylog version - Elasticsearch version Logfiles of Graylog and Elasticsearch might help too as without any it is just a wild guessing. But I think that your elasticsearch is not able to handle the load. ---
**Author:** ![n07n0w](https://avatars.discourse-cdn.com/v4/letter/n/f17d59/32.png) [@n07n0w](https://community.graylog.org/u/n07n0w)\ **Post date:** [December 21, 2017, 10:34am UTC](https://community.graylog.org/t/unprocessed-messages/3588/3 "2017-12-21T10:34:27Z")
Jan, thanks for reply! Graylog 2.3.2+3df951e Elasticsearch 5.6.5-1 13267 messages in output buffer, 5.06% utilized. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [December 21, 2017, 10:42am UTC](https://community.graylog.org/t/unprocessed-messages/3588/4 "2017-12-21T10:42:33Z")
> I think that your elasticsearch is not able to handle the load. ---
**Author:** ![n07n0w](https://avatars.discourse-cdn.com/v4/letter/n/f17d59/32.png) [@n07n0w](https://community.graylog.org/u/n07n0w)\ **Post date:** [December 21, 2017, 2:07pm UTC](https://community.graylog.org/t/unprocessed-messages/3588/5 "2017-12-21T14:07:56Z")
On all 4 nodes of elasticsearch LA 3-4 (with 8 cores) memory used on 50-60 percent. What else can be bad? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [December 21, 2017, 2:18pm UTC](https://community.graylog.org/t/unprocessed-messages/3588/6 "2017-12-21T14:18:02Z")
Did you checked the Health of your Cluster? Did you checked the Logfiles? Did you checked your Elasticsearch Metrics? All the above would you give information what is bad. ---
**Author:** ![jtkarvo](https://avatars.discourse-cdn.com/v4/letter/j/43a26b/32.png) [@jtkarvo](https://community.graylog.org/u/jtkarvo)\ **Post date:** [December 21, 2017, 5:44pm UTC](https://community.graylog.org/t/unprocessed-messages/3588/7 "2017-12-21T17:44:41Z")
Do you have \>90% CPU load on the Graylog server, when ? If not, you can still increase the number of processbuffer processors and see if that helps. Also, you have quite a lot of outputbuffer processors (8) for the outputbuffer size. You could get to 30000 msgs/s with just 3 outputbuffer processors, so you could move 5 of those to the processbuffer processors. ---
**Author:** ![n07n0w](https://avatars.discourse-cdn.com/v4/letter/n/f17d59/32.png) [@n07n0w](https://community.graylog.org/u/n07n0w)\ **Post date:** [December 25, 2017, 11:14am UTC](https://community.graylog.org/t/unprocessed-messages/3588/8 "2017-12-25T11:14:07Z")
Hi, I have \>90% CPU load on the Graylog server, LA 11 (on 8 core processor) outputbuffer processors is set on 3 now, maybe I need to add graylog-server nodes? Elasticsearch nodes is not overloaded, but numbers of unprocessed messages is still hight. ---
**Author:** ![jtkarvo](https://avatars.discourse-cdn.com/v4/letter/j/43a26b/32.png) [@jtkarvo](https://community.graylog.org/u/jtkarvo)\ **Post date:** [December 25, 2017, 11:53am UTC](https://community.graylog.org/t/unprocessed-messages/3588/9 "2017-12-25T11:53:33Z")
If you have checked that you don’t have problematic regexes in extractors or pipelines, then I’d say it is time to add more graylog nodes or processors to the existing node. But it would be good to check the regexes first. See e.g. [https://www.regular-expressions.info/catastrophic.html](https://www.regular-expressions.info/catastrophic.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [January 8, 2018, 11:53am UTC](https://community.graylog.org/t/unprocessed-messages/3588/10 "2018-01-08T11:53:33Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.