Hi @gsmith!
The CPU usage follows the message rate pretty closely, it is not constant. Here is an updated view of the last 48h.
If you are wondering, the excursion on graylog3 yesterday morning was me fiddling with the *_wait_strategy settings. The profile also followed message rate before the update, but usage as said was approximately 4x lower (curiously a similar amount as what seen in my 4.1.11 + Java 11 thread). If I understand correctly then you are suggesting that since it is not a constant high CPU usage, independent of the message rate, then it is expected that the *_wait_strategy have no role in it.
Regarding rolling back Java, as said, I can try. I usually do not rely on the global update-alternatives setting for Java, but explicitly configure it in /etc/default/graylog-server JAVA environment variable. As for the settings, I am also currently using the package defaults from /etc/default/graylog-server
# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"
# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"
# Fix for log4j CVE-2021-44228
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"
which results e.g. in
gian@graylog1:~$ cat /proc/$(pidof java)/cmdline | tr '\0' '\n'
/usr/lib/jvm/temurin-17-jdk-amd64/bin/java
-Xms4g
-Xmx4g
-XX:NewRatio=1
-server
-XX:+ResizeTLAB
-XX:-OmitStackTraceInFastThrow
-Djdk.tls.acknowledgeCloseNotify=true
-Dlog4j2.formatMsgNoLookups=true
-jar
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/usr/share/graylog-server/lib/sigar
-Dgraylog2.installation_source=deb
/usr/share/graylog-server/graylog.jar
server
-f
/etc/graylog/server/server.conf
-np
do you see any setting that needs tweaking? I do not have the need to change anything in Java’s keystore since the service is on a private network and I am not implementing TLS directly on graylog-server nodes.