Allow graylog in docker to use multiple cores

vasrem · June 5, 2018, 3:56pm

Hello,

I would like to ask if there is a way to allow java to spawn its threads to multiple cores when I run a dockerized graylog image.

Expected Behavior

I would like to use a specific number of cores that are available in my systems.

Current Behavior

Right now java uses only 2 cores while there are 10 in each node. Docker can identify that there are 10 cores in the system.

Possible Solution

Any JAVA_OPTS flag that could enable that feature

Steps to Reproduce (for bugs)

Context

There is a huge amount of messages arriving in the graylog cluster and the cluster is unable to process them. I have tried to adjust the [input,output,process]buffer_processors but this didn’t help. I think that the worst thing is that there are available resources but java doesn’t use them. This results to a very huge load (like 20-22), 100% usage at the first 2 cores of the system and also to the filling up of the journal.

Graylog is also installed in a different cluster without using docker and I’m not encountering this issue. The java threads are using all the cores.

Your Environment

processbuffer_processors = 5
outputbuffer_processors = 3
inputbuffer_processors = 2
message_journal_max_size = 20gb
GRAYLOG_SERVER_JAVA_OPTS=-Xms1g -Xmx4g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow

Graylog Version: 2.4.3-3
Elasticsearch Version: 5.6.7
MongoDB Version: 3.4.13
Operating System: Centos-release-7
Browser version:

jochen · June 5, 2018, 4:03pm

Which version of Docker are you using?
Which Docker image are you using?
What’s the complete configuration of the Docker container?
What’s the complete configuration of Graylog?

vasrem · June 6, 2018, 12:25pm

Docker version 18.03.0-ce
graylog2/server:2.4.3-3
https://pastebin.com/XWFBYxjK
https://pastebin.com/S2hEWdcd

I just replaced the passwords and some urls.

jochen · June 6, 2018, 12:38pm

Try using graylog/graylog:2.4.5-1.

The JVM used in the old graylog2/server Docker image doesn’t support Docker (see https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits). You’ll also have to update your GRAYLOG_SERVER_JAVA_OPTS environment variable accordingly.

Also see:

vasrem · June 6, 2018, 1:31pm

I have tried using the same JAVA_OPTS and using much simpler ones as well -Xms1g -Xmx4g -XX:+UnlockExperimentalVMOptions but it didnt work. Only 2 CPUs are used.

jochen · June 6, 2018, 2:33pm

Both is wrong. Please use the default value for GRAYLOG_SERVER_JAVA_OPTS as template for your customization.

github.com

Graylog2/graylog-docker/blob/2.4.5-1/Dockerfile#L48


      
            && wget -nv -O "/tmp/graylog-${GRAYLOG_VERSION}.tgz" "https://packages.graylog2.org/releases/graylog/graylog-${GRAYLOG_VERSION}.tgz" \
            && wget -nv -O "/tmp/graylog-${GRAYLOG_VERSION}.tgz.sha256.txt" "https://packages.graylog2.org/releases/graylog/graylog-${GRAYLOG_VERSION}.tgz.sha256.txt" \
            && cd /tmp \
            && sha256sum -c "/tmp/graylog-${GRAYLOG_VERSION}.tgz.sha256.txt" \
            && tar -xzf "/tmp/graylog-${GRAYLOG_VERSION}.tgz" --strip-components=1 -C /usr/share/graylog \
            && chown -R graylog:graylog /usr/share/graylog \
            && rm -f "/tmp/graylog-${GRAYLOG_VERSION}.tgz" \
            && apt-get update && apt-get -y install libcap2-bin \
            && setcap 'cap_net_bind_service=+ep' $JAVA_HOME/bin/java
          
          ENV GRAYLOG_SERVER_JAVA_OPTS "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow"
          ENV PATH /usr/share/graylog/bin:$PATH
          WORKDIR /usr/share/graylog
          
          RUN set -ex \
            && for path in \
              ./data/journal \
              ./data/log \
              ./data/config \
            ; do \
              mkdir -p "$path"; \

Are you sure that the Docker container is allowed to use more CPUs?
See Runtime options with Memory, CPUs, and GPUs | Docker Docs for details.

What’s the full output of the following command?

# docker inspect $CONTAINER_ID | grep -i cpu

Example:

$ docker inspect 654142f112ae | grep -i cpu
            "CpuShares": 0,
            "NanoCpus": 0,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "CpuCount": 0,
            "CpuPercent": 0,

vasrem · June 6, 2018, 2:45pm

Even the default + -Xms1g -Xmx4g isn’t working.

# docker inspect 0a81b1e53578 | grep -i cpu
            "CpuShares": 0,
            "NanoCpus": 0,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "CpuCount": 0,
            "CpuPercent": 0,

jochen · June 6, 2018, 2:58pm

That’s not the default for the new Docker image and as long as you don’t provide the complete and unredacted value of GRAYLOG_SERVER_JAVA_OPTS you’re using, we won’t be able to help you.

What’s the complete output of the following command?

# docker system info

vasrem · June 6, 2018, 3:04pm

So I’ve tried using
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow

and

-Xms1g -Xmx4g -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow

but both are not working.

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.2.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 7.693GiB
Name: XXXXXXXXXXXXXXXXXXXXXXXX
ID: XXXXXXXXXXXXXXXXXXXXXXXX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

jochen · June 6, 2018, 3:47pm

Please run the Docker image groovy:2.5.0-jre8 and post the output when executing the expression Runtime.runtime.availableProcessors().

Example:

$ docker run -it groovy:2.5.0-jre8
Jun 06, 2018 3:46:46 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Groovy Shell (2.5.0, JVM: 1.8.0_171)
Type ':help' or ':h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> Runtime.runtime.availableProcessors()
===> 4
groovy:000> :exit

vasrem · June 7, 2018, 6:49am

I get this response. I guess something is wrong with the docker engine (?)

# docker run -it groovy:2.5.0-jre8
Unable to find image 'groovy:2.5.0-jre8' locally
2.5.0-jre8: Pulling from library/groovy
cc1a78bfd46b: Already exists
d2c05365ee2a: Pull complete
231cb0e216d3: Pull complete
e8912f9d0ce2: Pull complete
9bafe362f99b: Pull complete
28b0652112a6: Pull complete
da23e1e20eae: Pull complete
9d809d99b239: Pull complete
7ff0806e176a: Pull complete
b0ab2872bc72: Pull complete
Digest: sha256:130410986a5ef8db430f7f89c1397c41d11347a8ecc47a38e89155896c35d35b
Status: Downloaded newer image for groovy:2.5.0-jre8
Jun 07, 2018 6:48:04 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Groovy Shell (2.5.0, JVM: 1.8.0_171)
Type ':help' or ':h' for help.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> Runtime.runtime.availableProcessors()
===> 2
groovy:000>

jochen · June 7, 2018, 7:41am

Try explicitly assigning the number of CPU cores you want to use in the container and try again:

vasrem · June 7, 2018, 8:20am

Thanks for all the help. I have found the issue. It had to do with systemd and not with java or dockerd. It was my mistake. I changed the resources of the VMs but didn’t restart dockerd to refresh the cgroup limits.

system · June 21, 2018, 8:20am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Stream processing disabled / not using all ressources Graylog Central (peer support)	4	423	September 4, 2018
High CPU usage after 3.0 -> 3.1 upgrade Graylog Central (peer support)	5	548	September 4, 2019
Graylog heap size on Docker Graylog Central (peer support)	12	8021	October 13, 2017
Multinode in graylog Graylog Central (peer support)	4	399	January 21, 2020
Graylog docker image with openjdk11? Graylog Central (peer support)	4	425	March 22, 2021