Graylog 4.1.x high CPU usage after updating for log4j

Hi again!

I can confirm that AFAIK switching to Java 8 solved the issue. Today I switched all the other servers too, restoring CPU usage to levels that seem comparable with the ones I was accustomed to.

I cannot say unfortunately, maybe someone from Graylog may comment on this. Given that all started with the Graylog update (not with an OpenJDK update), my impression as an outsider is that it may well be that the Graylog’s 4.1 support for Java 11 was never more than experimental (as stated in the docs), and some code update between 4.1.1 and 4.1.9 took with it something that made some performance-affecting incompatibility rear its ugly head. Logs were hinting at the possibility that there may be something at play, and maybe everything was tested on Java 8, so it was not detected until it hit a real world installation hard enough. I probably should have followed the installation guide more thoroughly, and if someone from Graylog finds it useful, I would suggest to consider removing the bits about Java 11 support in the archived docs, or making the risks of running it more evident.

It was a new find for me too. I came across them while looking for AdoptOpenJDK. which I also remembered the initiative being one of the sanest ways to easily install a “tested” OpenJDK binary distribution from some years ago. From what I can tell, the AdoptOpenJDK project simply went under the umbrella of the Eclipse foundation and rebranded because of shared interests and benefits. Temurin is the new name of the project and distribution, Adoptium is the higher level project handling Temurin, both are inside the Eclipse projects ecosystem. I am however completely ignorant on the matter apart from this!

As said, I chose them as an alternative because unless I lost something, Debian buster has only one true alternative, which is openjdk-11-jre-headless. There is an additional “zero” variant, which does not seem suited to my use case, and also a non-free JRE 8 from NVIDIA which is also a no-go for me (openly stated as obsolete for compatibility for some NVIDIA applications, and non-free). Adoptium unfortunately does not yet provide packages so I went for the ugly solution of dropping the tarball on the filesystem manually.

Regarding the method for the Java switch, Debian (and other distros) would certainly like users use the alternatives framework as you suggest. I think that the framework is the way to go to provide a system-wide or default setting, and is generally well-integraded with distributions that provide multiple alternative Java versions, but I saw no reason to shy away from providing a specific JRE to a specific application for performance or stability benefits. If I wanted to hook my manual Temurin installation up to the alternatives framework, I certainly could with some busywork, but I did not see any advantage in my case since Graylog is the only java application on the servers. Fortunately /etc/default/graylog-server provides a way to directly select the java binary with the JAVA variable, thus in my case I simply set

JAVA=/opt/temurin/jre8/bin/java

and Graylog started using that instead of the default, alternatives-managed /usr/bin/java, while I still have the Debian-provided OpenJDK 11 as a more easily maintained default.

I am marking the topic as solved! It has been a strange ride, but all is well that ends well. Thank you for you continued support!

3 Likes