# Sudden performance downgrade and search error **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [July 27, 2020, 11:28am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547 "2020-07-27T11:28:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bbfunde](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bbfunde](https://community.graylog.org/u/bbfunde)\ **Post date:** [July 27, 2020, 11:28am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/1 "2020-07-27T11:28:44Z")
Hi everyone, _ **graylog version: v3.3.2+ade4779** _ _ **VM resources: 16 vCPU + 96GB RAM** _ _ **One node** _ Recently the dashboard performance experiences downgrade so I tried to fine-tune the settings by increasing the graylog JVM heap space to 16GB and the buffer processors settings in graylog server.conf: _processbuffer\_processors = 16_ _outputbuffer\_processors = 9_ **Afterwards, the dashboard took a very long time to search and load the widgets and finally timed out.** ![graylog_img1](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/be7253e93bc05c095387d7c9d9d5eeb818186b6f.png) ![graylog_img3](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/7/7c325a25b6d92ab5360a2c89744a04d1bb588fba.png) I have tried to stop all inputs except one, disable all pipelines, and delete all extractors created shortly before this happened. But don’t help. I found out that there are tons of unprocessed messages and the process and output buffer are all exhausted. Maybe this is the reason? And if it is, can anyone suggest a way to alleviate the pain? 🥺 ![graylog_img2](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/4/458447795fee09365d876ba526d2f0c0ac64759e.png) Thank you in advance. ---
**Author:** ![lcosta](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/lcosta/32/6109_2.png) [@lcosta](https://community.graylog.org/u/lcosta)\ **Post date:** [July 28, 2020, 2:07pm UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/2 "2020-07-28T14:07:04Z")
Did you check Elasticsearch log? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [July 29, 2020, 6:21am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/3 "2020-07-29T06:21:45Z")
he @bbfunde when you have added those 16 cores to the VM that hosts Elasticsearch AND Graylog it is very likely that both are fighting for the CPUs … You should limit the number of cores that are available for Elasticsearch and not overcommit the processors on Graylog. The picture that you have output buffer filled is the indicator that your elasticsearch is needing to long to index the messages and return/free the output processor of Graylog, what causes a jam in the working queue. ---
**Author:** ![bbfunde](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bbfunde](https://community.graylog.org/u/bbfunde)\ **Post date:** [July 29, 2020, 8:04am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/4 "2020-07-29T08:04:51Z")
thanks for the reply, I think I have found the cause following Jan’s advice below. ---
**Author:** ![bbfunde](https://avatars.discourse-cdn.com/v4/letter/b/f4b2a3/32.png) [@bbfunde](https://community.graylog.org/u/bbfunde)\ **Post date:** [July 29, 2020, 8:13am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/5 "2020-07-29T08:13:11Z")
Thanks Jan for the advice. I have done some research on how to optimize the Elasticsearch and adjust some settings including memory and swap file. The issue is seemingly gone for days now. **But I can’t find where to restrict the processor number for Elasticsearch.** I can’t find the setting in Elasticsearch.yml (I am running it as a service). Can only find some running it in Docker or from command line. Can you help giving me a reference on that. 🙏 Thank you a lot. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [July 29, 2020, 9:51am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/6 "2020-07-29T09:51:54Z")
[https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html#processors](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html#processors) ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [August 12, 2020, 9:51am UTC](https://community.graylog.org/t/sudden-performance-downgrade-and-search-error/16547/7 "2020-08-12T09:51:55Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.