Configuration Issue

Graylog Central (peer support)
1

I am very much a Graylog noob and have followed the instructions to the best of my ability. Graylog is working, it is just over whelmed with logs. I am not sure what to set the configs to to make the most out of the server. What details should I post here in order to get help with the configs?

Graylog Server Specs:
RHEL 8.3
Graylog 4.0.2
Elasticsearch 6.8.13
JVM 1.8.0
VMware 6.7
12 CPUs
24 GB Memory
500 GB HDD

GL_Deatils
GL_Deatils1195×554 123 KB

2

Just a cursory glance leads me to think that Elasticsearch doesn’t have enough resources to chew through your messages. What do you have the heap set for in ES (/etc/default/elasticsearch in deb, /etc/sysconfig/elasticsearch in RHEL) and Graylog (/etc/default/graylog in deb, /etc/sysconfig/graylog in RHEL) ?

3

Elasticsearch

#Path to the java executable.
JAVA=/usr/bin/java

#Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS=“-Xms16g -Xmx16g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow”

#Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS ->Djdk.tls.acknowledgeCloseNotify=true”

#Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)
GRAYLOG_SERVER_ARGS=“”

#Program that will be used to wrap the graylog-server command. Useful to
#support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=“”

Graylog

#Path to the java executable.
JAVA=/usr/bin/java

#Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS=“-Xms16g -Xmx16g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow”

#Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true”

#Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)
GRAYLOG_SERVER_ARGS=“”

#Program that will be used to wrap the graylog-server command. Useful to
#support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=“”

4

So, collectively you have 32GB of heap assigned on a box with 24GB RAM?

1 Like
5

Just an assumption, that would be bad correct? I need to increase the virtual memory to 32GB?

6

I just realized that I accidently posted the Graylog config twice. Here is Elasticsearch

################################
#Elasticsearch
################################

#Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch

#Elasticsearch Java path
#JAVA_HOME=

#Elasticsearch configuration directory
ES_PATH_CONF=/etc/elasticsearch

#Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

#Additional Java OPTS
#ES_JAVA_OPTS=

#Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
#Elasticsearch service
################################

#SysV init.d

#The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5

################################
#System properties
################################

#Specifies the maximum file descriptor number that can be opened by this process
#When using Systemd, this setting is ignored and the LimitNOFILE defined in
#/usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65535

#The maximum number of bytes of memory that may be locked into RAM
#Set to “unlimited” if you use the ‘bootstrap.memory_lock: true’ option
#in elasticsearch.yml.
#When using systemd, LimitMEMLOCK must be set in a unit file such as
#/etc/systemd/system/elasticsearch.service.d/override.conf.
#MAX_LOCKED_MEMORY=unlimited

#Maximum number of VMA (Virtual Memory Areas) a process can own
#When using Systemd, this setting is ignored and the ‘vm.max_map_count’
#property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144

7

I thought the config for elastisearch is in /etc/sysconfig/elastisearch.

ES_HEAP_SIZE should be half of available system memory running only es on the system.

8

I have not modified the Elasticsearch config file. Everything is defaulted.

9

Ah! Ok. That makes sense. So, you presently have 16GB assigned to Graylog, but Elastic has nothing in ES_JAVA_OPTS, which IIRC, means that it takes a default of 1GB. You can change that with the following:

# Additional Java OPTS
ES_JAVA_OPTS="-Xms3g -Xmx3g"

Though since you’ve clearly got more RAM, I’d bump it up closer to something like 6GB for ES.

FWIW, you might consider tuning this a bit more finely and maybe end up giving 10G in heap each for Elastic and Graylog. Unless you’re seeing that Graylog is having trouble keeping up with processing messages, in which case you might benefit more from scaling Graylog out than up.

10

That instantly resolved the issue. Thank you very much.

1 Like
11

Glad to hear it, and more than happy to help!

1 Like
12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.