In my Graylog cluster, the Java heap size is set with a minimum of 2 GB and a maximum of 12 GB for the Graylog nodes. However sometimes it exceeded the threshold limit, causing the nodes to restart. logs are loss then after manually restart then it works properly
Are you able to share any applicable log entries from Graylog’s server.log when this happens?
Are you collecting metrics via prometheus? I’m curious what your operating system metrics look like for memory utilization. Are you having memory contention issues causes an OOM (out of memory) event and the Graylog process is killed? Or is graylog itself restarting?
Another important thing is that the heap for Graylog should not exceed 50% of available system ram.
I’d like to understand more about your memory utilization of your server and if the issue is related to memory contention or the OOM killer.
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2024-11-25 15:03:56 IST; 1 weeks 2 days ago
Docs: http://docs.graylog.org/
Main PID: 14919 (graylog-server)
Tasks: 4547 (limit: 4653)
CGroup: /system.slice/graylog-server.service
├─14919 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─14971 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeClos
Dec 04 14:52:32 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-18” java.lang.OutOfMemoryError: Java heap space
Dec 04 15:27:30 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-0” java.lang.OutOfMemoryError: Java heap space
Dec 04 15:29:32 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-15” java.lang.OutOfMemoryError: Java heap space
Dec 04 15:29:32 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-22” java.lang.OutOfMemoryError: Java heap space
Dec 04 16:04:51 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-9” SLF4J: Failed toString() invocation on an object of ty
Dec 04 16:04:51 ubuntu11 graylog-server[14919]: Reported exception:
Dec 04 16:04:52 ubuntu11 graylog-server[14919]: java.lang.OutOfMemoryError: Java heap space
Dec 04 16:04:52 ubuntu11 graylog-server[14919]: java.lang.OutOfMemoryError: Java heap space
Dec 04 16:26:52 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-7” java.lang.OutOfMemoryError: Java heap space
Dec 04 16:27:12 ubuntu11 graylog-server[14919]: Exception in thread “netty-transport-20” java.lang.OutOfMemoryError: Java heap space
lines 1-20/20 (END)
Interesting, according to /usr/bin/java -Xms1g -Xmx1g Graylog is only configured to use 1g of heap. Can you verify how you configured Graylog heap?
Should be set using JVM settings
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.