# Process buffer and output buffer are full. Journal over the allowed size. No messages written to elastic **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** elastic\ **Created:** [October 2, 2024, 10:34am UTC](https://community.graylog.org/t/process-buffer-and-output-buffer-are-full-journal-over-the-allowed-size-no-messages-written-to-elastic/33737 "2024-10-02T10:34:37Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![chimi](https://avatars.discourse-cdn.com/v4/letter/c/73ab20/32.png) [@chimi](https://community.graylog.org/u/chimi)\ **Post date:** [October 2, 2024, 10:34am UTC](https://community.graylog.org/t/process-buffer-and-output-buffer-are-full-journal-over-the-allowed-size-no-messages-written-to-elastic/33737/1 "2024-10-02T10:34:37Z")
1. Describe your incident: I’m running 4 instances of Graylog nodes 5.0.5 managed by load balancer and 5 instances of ElasticSearch. Those four nodes are not sending messages to elasticsearch and there are a lot of messages that saturates the journal. In server.log there are a lot of this error messages: ERROR [Messages] Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred:]; nested: IOException[Connection reset]; nested: SocketException[Connection reset];, errorDetails=}, retrying (attempt #38) All hosts of elastic and graylog clusters are on the same subnet and there is nothing to prevent communication between them even testing it manually there do not seem to be any problems. The infrastructure has been running smoothly for months and suddenly this problem has arisen. Instead in elasticsearch’s side there aren’t evident issues, and the resource consumption is low. 1. Describe your environment: Graylog nodes details: CPU Architecture: x86\_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7J13 64-Core Processor Stepping: 1 CPU MHz: 2445.392 BogoMIPS: 4890.78 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 64K L1i cache: 64K L2 cache: 512K L3 cache: 16384K NUMA node0 CPU(s): 0-15 OS version NAME=“Oracle Linux Server” VERSION=“8.8” ID=“ol” ID\_LIKE=“fedora” VARIANT=“Server” VARIANT\_ID=“server” VERSION\_ID=“8.8” PLATFORM\_ID=“platform:el8” PRETTY\_NAME=“Oracle Linux Server 8.8” ANSI\_COLOR=“0;31” CPE\_NAME=“cpe:/o:oracle:linux:8:8:server” Elasticsearch Version:7.10.2 Graylog Version:5.0.5 GRAYLOG graylog.conf is\_leader = true node\_id\_file = /appl/graylog/graylog-5.0.5-linux-x64/node\_id password\_secret = xxxx root\_password\_sha2 = xxxx root\_timezone = America/Sao\_Paulo bin\_dir = /appl/graylog/graylog-5.0.5-linux-x64/bin data\_dir = /appl/graylog/graylog-5.0.5-linux-x64/data plugin\_dir = /appl/graylog/graylog-5.0.5-linux-x64/plugin http\_bind\_address = node\_ip\_addr:9000 http\_enable\_tls = true http\_tls\_cert\_file = /appl/graylog/cert.pem http\_tls\_key\_file = /appl/graylog/pkcs8-encrypted.pem http\_tls\_key\_password = pwd stream\_aware\_field\_types=false elasticsearch\_hosts = host1:9200,host2:9200,host3:9200,host4:9200,host5:9200 rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 20000000 elasticsearch\_max\_number\_of\_indices = 20 retention\_strategy = delete elasticsearch\_shards = 3 elasticsearch\_replicas = 0 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_analyzer = standard output\_batch\_size = 1000 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 7 outputbuffer\_processors = 4 processor\_wait\_strategy = blocking ring\_size = 65536 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 3 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir = data/journal message\_journal\_max\_age = 24h message\_journal\_max\_size = 10gb lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb://host1:27017,host2:27017,host3:27017,host4:27017/graylog?replicaSet=rs0 mongodb\_max\_connections = 1000 ELASTICSEARCH CONFIG cluster.name: elasticsearch\_cluster node.name: hostname node.attr.rack: r1 path.data: /appl/elasticsearch/elasticsearch-7.10.2/data path.logs: /appl/elasticsearch/elasticsearch-7.10.2/log network.host: ip\_add discovery.seed\_hosts: [“host1”, “host2”, “host3”,“host4”,“host5”] cluster.initial\_master\_nodes: [“host1-1”, “host2-2”, “host3-3”] gateway.recover\_after\_nodes: 2 action.auto\_create\_index: false opendistro\_security.ssl.transport.pemcert\_filepath: esnode.pem opendistro\_security.ssl.transport.pemkey\_filepath: esnode-key.pem opendistro\_security.ssl.transport.pemtrustedcas\_filepath: root-ca.pem opendistro\_security.ssl.transport.enforce\_hostname\_verification: false opendistro\_security.ssl.http.enabled: false opendistro\_security.ssl.http.pemcert\_filepath: esnode.pem opendistro\_security.ssl.http.pemkey\_filepath: esnode-key.pem opendistro\_security.ssl.http.pemtrustedcas\_filepath: root-ca.pem opendistro\_security.allow\_unsafe\_democertificates: true opendistro\_security.allow\_default\_init\_securityindex: true opendistro\_security.authcz.admin\_dn: - CN=kirk,OU=client,O=client,L=test, C=de 1. What steps have you already taken to try and solve the problem? I tried to raise the number of processors allocated, processbuffer\_processors = 7, outputbuffer\_processors = 4, inputbuffer\_processor=3 but nothing changed. I restarted several time the graylog services, seems to work initially but now the situation seems blocked. I tried also to increase Field type refresh interval from 5 s to 30 s Any idea on what to troubleshoot in order to solve this issue? ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [October 3, 2024, 10:38am UTC](https://community.graylog.org/t/process-buffer-and-output-buffer-are-full-journal-over-the-allowed-size-no-messages-written-to-elastic/33737/2 "2024-10-03T10:38:17Z")
Hello @chimi Does the the cluster health api call return anything of interest? Are you sure there is nothing is any of the logs on your Opensearch hosts? ---
**Author:** ![chimi](https://avatars.discourse-cdn.com/v4/letter/c/73ab20/32.png) [@chimi](https://community.graylog.org/u/chimi)\ **Post date:** [October 4, 2024, 12:29pm UTC](https://community.graylog.org/t/process-buffer-and-output-buffer-are-full-journal-over-the-allowed-size-no-messages-written-to-elastic/33737/3 "2024-10-04T12:29:46Z")
Hello @Wine_Merchant There aren’t usefull logs into the Elasticsearch hosts, and also the cluster health api return alway green as status and nothing else interesting. The problem seems to be on the communication between Graylog and Elastic on the Graylog side. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [October 18, 2024, 12:30pm UTC](https://community.graylog.org/t/process-buffer-and-output-buffer-are-full-journal-over-the-allowed-size-no-messages-written-to-elastic/33737/4 "2024-10-18T12:30:24Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.