# Buffer Configuration Issue

**URL:** <https://community.graylog.org/t/buffer-configuration-issue/20109>\
**Category:** Graylog Central (peer support)\
**Created:** [June 7, 2021, 1:53pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109 "2021-06-07T13:53:59Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![monarch684](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@monarch684](https://community.graylog.org/u/monarch684)\
**Post date:** [June 7, 2021, 1:54pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/1 "2021-06-07T13:54:00Z")

</div>

My Graylog server recently started having issues with the Process and Output buffers being full 100% of the time. I increased the CPU count on the VM and modified the ‘processbuffer\_processors’ and ‘outputbuffer\_processors’ accordingly to no avail. I know I need to adjust something else, I am not sure what that something else is.

## Server Config

RHEL 8.3  
Graylog 4.0.5+d95b909, codename _Noir_  
Elasticsearch 6.8.15  
Current Hardware  
24 CPUs  
32 GB Memory  
3TB space

## Graylog Config

> is\_master = true  
> node\_id\_file = /etc/graylog/server/node-id  
> password\_secret =  
> root\_password\_sha2 =  
> root\_timezone = America/Chicago  
> bin\_dir = /usr/share/graylog-server/bin  
> data\_dir = /var/lib/graylog-server  
> plugin\_dir = /usr/share/graylog-server/plugin  
> http\_bind\_address =  
> http\_bind\_address =  
> rotation\_strategy = count  
> elasticsearch\_max\_docs\_per\_index = 20000000  
> elasticsearch\_max\_number\_of\_indices = 20  
> retention\_strategy = delete  
> elasticsearch\_shards = 4  
> elasticsearch\_replicas = 0  
> elasticsearch\_index\_prefix = graylog  
> allow\_leading\_wildcard\_searches = false  
> allow\_highlighting = false  
> elasticsearch\_analyzer = standard  
> output\_batch\_size = 5000  
> output\_flush\_interval = 1  
> output\_fault\_count\_threshold = 5  
> output\_fault\_penalty\_seconds = 30  
> processbuffer\_processors = 8  
> outputbuffer\_processors = 4  
> processor\_wait\_strategy = blocking  
> ring\_size = 65536  
> inputbuffer\_ring\_size = 65536  
> inputbuffer\_processors = 4  
> inputbuffer\_wait\_strategy = blocking  
> message\_journal\_enabled = true  
> message\_journal\_dir = /var/lib/graylog-server/journal  
> message\_journal\_max\_size = 15gb  
> lb\_recognition\_period\_seconds = 3  
> mongodb\_uri = mongodb://localhost/graylog  
> mongodb\_max\_connections = 1000  
> mongodb\_threads\_allowed\_to\_block\_multiplier = 5  
> transport\_email\_enabled = true  
> transport\_email\_hostname =  
> transport\_email\_port = 25  
> transport\_email\_subject\_prefix = [graylog]  
> transport\_email\_from\_email =  
> proxied\_requests\_thread\_pool\_size = 32  
> versionchecks = false

## Elasticsearch Config

> cluster.name: graylog  
> path.data: /home/elasticsearch/elasticsearch  
> path.logs: /home/elasticsearch/log  
> action.auto\_create\_index: false

As you can see, I do not have a complicated setup. If there are any other logs that need to be displayed, just let me know.

---

<div class="post-metadata">

**Author:** ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)\
**Post date:** [June 7, 2021, 6:32pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/2 "2021-06-07T18:32:25Z")

</div>

If your output buffer and process buffer are both at 100%, then there’s a resourcing issue (likely around heap). Have you tried increasing Graylog’s heap or Elasticsearch’s heap at all?

---

<div class="post-metadata">

**Author:** ![monarch684](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@monarch684](https://community.graylog.org/u/monarch684)\
**Post date:** [June 7, 2021, 6:56pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/3 "2021-06-07T18:56:55Z")

</div>

I have. The Graylog heap is set to 8GB and I just up the Elasticsearch to 8GB. I didn’t realize that it was set to 1GB.

Graylog

```auto
# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUn$

# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=""

```

Elasticsearch

```auto
## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms8g
-Xmx8g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10

## optimizations

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow

# enable helpful NullPointerExceptions (https://openjdk.java.net/jeps/358), if
# they are supported
14-:-XX:+ShowCodeDetailsInExceptionMessages

# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

# temporary workaround for C2 bug with JDK 10 on hardware with AVX-512
10-:-XX:UseAVX=2

```

---

<div class="post-metadata">

**Author:** ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)\
**Post date:** [June 7, 2021, 7:01pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/4 "2021-06-07T19:01:03Z")

</div>

So, both Graylog and Elasticsearch take their settings from the environment variable files found in either `/etc/default/{graylog-server,elasticsearch}` or `/etc/sysconfig/{graylog-server,elasticsearch}`, depending on the distro. For Graylog, the file looks like

```auto
cat /etc/default/graylog-server
# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"

# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=""

```

You’d change the heap in the `GRAYLOG_SERVER_JAVA_OPTS` variable.

---

<div class="post-metadata">

**Author:** ![monarch684](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@monarch684](https://community.graylog.org/u/monarch684)\
**Post date:** [June 7, 2021, 7:07pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/5 "2021-06-07T19:07:04Z")

</div>

Not sure why my post is formatted this way. Apologies.

---

<div class="post-metadata">

**Author:** ![aaronsachs](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aaronsachs/32/7180_2.png) [@aaronsachs](https://community.graylog.org/u/aaronsachs)\
**Post date:** [June 7, 2021, 7:09pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/6 "2021-06-07T19:09:47Z")

</div>

No worries–discourse uses Markdown, so `#` get interpreted as header levels. Surrounding any configs with ``` will do the trick to keep things from being oddly formatted.

Since both are now using 8GB, are you still seeing a backlog of messages?

---

<div class="post-metadata">

**Author:** ![monarch684](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@monarch684](https://community.graylog.org/u/monarch684)\
**Post date:** [June 7, 2021, 7:49pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/7 "2021-06-07T19:49:17Z")

</div>

Uh-oh…I think in making some of these changes I broke Graylog. Gotta put this convo on pause so I can go see what I broke.

---

<div class="post-metadata">

**Author:** ![monarch684](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@monarch684](https://community.graylog.org/u/monarch684)\
**Post date:** [June 17, 2021, 1:43pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/8 "2021-06-17T13:43:24Z")

</div>

So I broke the server enough to where I had to rebuild. Buffer is not a issue any more.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [July 1, 2021, 1:43pm UTC](https://community.graylog.org/t/buffer-configuration-issue/20109/9 "2021-07-01T13:43:54Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.