We recently upgraded Graylog from version 3.3.14 to version 4.1.6. As such the server.conf file changed the TLS settings to only use TLSv1.2 and TLSv1.3 as the defaults. However, enabling the latter is causing the server to drop after 15-30mins.
server.conf file:
# The allowed TLS protocols for system wide TLS enabled servers. (e.g. message inputs, http interface)
# Setting this to an empty value, leaves it up to system libraries and the used JDK to chose a default.
# Default: TLSv1.2,TLSv1.3 (might be automatically adjusted to protocols supported by the JDK)
enabled_tls_protocols = TLSv1.2,TLSv1.3
Error output in the log file:
2021-10-31T18:58:19.363Z WARN [ProxiedResource] Unable to call https://graylog.companyname.com:9000/api/system/metrics/multiple on node <b20
c8480-0cd9-43b7-afcf-9212d536ac47>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid ce
rtification path to requested target
Description of steps you’ve taken to attempt to solve the issue
Restarting the Graylog server will bring it back up, but 15-30mins later it will drop again.
Temporary resolution has been to go into the server.conf file and remove TLSv1.3 from being enabled, which ensures that Graylog remains up and stable.
Ah ok. So the reason is that JDK 8 & TLS 1.3 don’t play nice together. You’ll either have to force the allowed ciphers to 1.2, or use Java 11. See enabled_tls_protocols in server.conf - Configuring Graylog. It’s definitely something that we could better surface in the docs.
That makes sense, thank you! Is there a recommended procedure/documentation in how to go about upgrading to Java 11 so as not to accidentally break Graylog?
I found this post (Upgrading from openjdk 8 to 11) about someone that went from version 1.8 to 11 and they mentioned an issue with changing garbage collection settings. Is this the main consideration needed or would other areas need to be changed to ensure a smooth transition?
Since Java 11 is supported, you should just be able to upgrade and use it. I’m not familiar with any in-depth JVM settings that need to be changed, but if you’re concerned about garbage collection using G1 garbage collection, you can refer to Best practice for JVM Tuning with G1 GC - Knowledge - BackStage for some useful information about using it explicitly. Other than that, some other folks in the community might have some additional input.
