Allow graylog in docker to use multiple cores


(Vasilis Remmas) #1

Hello,

I would like to ask if there is a way to allow java to spawn its threads to multiple cores when I run a dockerized graylog image.

Expected Behavior

I would like to use a specific number of cores that are available in my systems.

Current Behavior

Right now java uses only 2 cores while there are 10 in each node. Docker can identify that there are 10 cores in the system.

Possible Solution

Any JAVA_OPTS flag that could enable that feature

Steps to Reproduce (for bugs)

Context

There is a huge amount of messages arriving in the graylog cluster and the cluster is unable to process them. I have tried to adjust the [input,output,process]buffer_processors but this didn’t help. I think that the worst thing is that there are available resources but java doesn’t use them. This results to a very huge load (like 20-22), 100% usage at the first 2 cores of the system and also to the filling up of the journal.

Graylog is also installed in a different cluster without using docker and I’m not encountering this issue. The java threads are using all the cores.

Your Environment

  • processbuffer_processors = 5
  • outputbuffer_processors = 3
  • inputbuffer_processors = 2
  • message_journal_max_size = 20gb
  • GRAYLOG_SERVER_JAVA_OPTS=-Xms1g -Xmx4g -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow
  • Graylog Version: 2.4.3-3
  • Elasticsearch Version: 5.6.7
  • MongoDB Version: 3.4.13
  • Operating System: Centos-release-7
  • Browser version:

(Jochen) #2

Which version of Docker are you using?
Which Docker image are you using?
What’s the complete configuration of the Docker container?
What’s the complete configuration of Graylog?


(Vasilis Remmas) #3

I just replaced the passwords and some urls.


(Jochen) #4

Try using graylog/graylog:2.4.5-1.

The JVM used in the old graylog2/server Docker image doesn’t support Docker (see https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits). You’ll also have to update your GRAYLOG_SERVER_JAVA_OPTS environment variable accordingly.

Also see:


(Vasilis Remmas) #5

I have tried using the same JAVA_OPTS and using much simpler ones as well -Xms1g -Xmx4g -XX:+UnlockExperimentalVMOptions but it didnt work. Only 2 CPUs are used.


(Jochen) #6

Both is wrong. Please use the default value for GRAYLOG_SERVER_JAVA_OPTS as template for your customization.

Are you sure that the Docker container is allowed to use more CPUs?
See https://docs.docker.com/config/containers/resource_constraints/#cpu for details.

What’s the full output of the following command?

# docker inspect $CONTAINER_ID | grep -i cpu

Example:

$ docker inspect 654142f112ae | grep -i cpu
            "CpuShares": 0,
            "NanoCpus": 0,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "CpuCount": 0,
            "CpuPercent": 0,

(Vasilis Remmas) #7

Even the default + -Xms1g -Xmx4g isn’t working.

# docker inspect 0a81b1e53578 | grep -i cpu
            "CpuShares": 0,
            "NanoCpus": 0,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "CpuCount": 0,
            "CpuPercent": 0,

(Jochen) #8

That’s not the default for the new Docker image and as long as you don’t provide the complete and unredacted value of GRAYLOG_SERVER_JAVA_OPTS you’re using, we won’t be able to help you.

What’s the complete output of the following command?

# docker system info

(Vasilis Remmas) #9

So I’ve tried using
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow

and

-Xms1g -Xmx4g -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow

but both are not working.

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.2.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 7.693GiB
Name: XXXXXXXXXXXXXXXXXXXXXXXX
ID: XXXXXXXXXXXXXXXXXXXXXXXX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

(Jochen) #10

Please run the Docker image groovy:2.5.0-jre8 and post the output when executing the expression Runtime.runtime.availableProcessors().

Example:

$ docker run -it groovy:2.5.0-jre8
Jun 06, 2018 3:46:46 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Groovy Shell (2.5.0, JVM: 1.8.0_171)
Type ':help' or ':h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> Runtime.runtime.availableProcessors()
===> 4
groovy:000> :exit

(Vasilis Remmas) #11

I get this response. I guess something is wrong with the docker engine (?)

# docker run -it groovy:2.5.0-jre8
Unable to find image 'groovy:2.5.0-jre8' locally
2.5.0-jre8: Pulling from library/groovy
cc1a78bfd46b: Already exists
d2c05365ee2a: Pull complete
231cb0e216d3: Pull complete
e8912f9d0ce2: Pull complete
9bafe362f99b: Pull complete
28b0652112a6: Pull complete
da23e1e20eae: Pull complete
9d809d99b239: Pull complete
7ff0806e176a: Pull complete
b0ab2872bc72: Pull complete
Digest: sha256:130410986a5ef8db430f7f89c1397c41d11347a8ecc47a38e89155896c35d35b
Status: Downloaded newer image for groovy:2.5.0-jre8
Jun 07, 2018 6:48:04 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Groovy Shell (2.5.0, JVM: 1.8.0_171)
Type ':help' or ':h' for help.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> Runtime.runtime.availableProcessors()
===> 2
groovy:000>

(Jochen) #12

Try explicitly assigning the number of CPU cores you want to use in the container and try again:


(Vasilis Remmas) #13

Thanks for all the help. I have found the issue. It had to do with systemd and not with java or dockerd. It was my mistake. I changed the resources of the VMs but didn’t restart dockerd to refresh the cgroup limits.


(system) #14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.