HTTPS login working but need help adding cert to JVM trust store

Graylog Central (peer support)
1

I wish for my graylog server to work with https so I read the below instructions…

http://docs.graylog.org/en/2.2/pages/configuration/https.html

I am able to login to the web interface using https but the Cisco input (from https://marketplace.graylog.org/addons/92c7e8b5-8f9b-44c1-9b36-7bc41685d2fe) is no longer working. I am wondering if it is because I could not figure out how to add the cert to the JVM trust store.

From the instructions…

"In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If you’ve been using another password to encrypt the JVM trust store than the default changeit, you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret.

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore and (optionally) javax.net.ssl.trustStorePassword system properties."

How do I get the start the graylog JVM with the trustStore parameter?

2

Errors I am receiving…

2017-05-02T22:40:53.625-04:00 WARN  [ProxiedResource] Unable to call https://10.16.1.90:9000/api/system/metrics/multiple on node <b6c85752-7456-4248-a5f5-2693c1fd94f7>
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:1.8.0_131]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:242) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:200) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.buildConnection(RealConnection.java:174) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:114) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:196) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:132) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:101) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_131]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:1.8.0_131]
	... 36 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_131]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_131]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_131]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:1.8.0_131]
	... 36 more
2017-05-02T22:40:57.096-04:00 WARN  [ProxiedResource] Unable to call https://10.16.1.90:9000/api/system/metrics/multiple on node <b6c85752-7456-4248-a5f5-2693c1fd94f7>
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:1.8.0_131]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:1.8.0_131]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:1.8.0_131]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:242) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:200) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.buildConnection(RealConnection.java:174) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:114) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:196) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:132) ~[graylog.jar:?]
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:101) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179) ~[graylog.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:63) ~[graylog.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_131]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:1.8.0_131]
	... 36 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_131]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_131]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[?:1.8.0_131]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[?:1.8.0_131]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:1.8.0_131]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:1.8.0_131]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:1.8.0_131]
	... 36 more
3

if your certificate is in dir -where you want- you can use the following commands (check parameters and substitute your own file names)

cp -a /usr/lib/jvm/jre-1.8.0-openjdk/lib/security/cacerts /-where you want-/cacerts.jks 
keytool -importcert -keystore /-where you want-/cacerts.jks -storepass changeit -alias graylog-CA-signed -file /-where you want-/graylog-cert.pem

Then set the Java VM to use your modified keystore file with the parameter you know already.

4

https://github.com/3vi1john/graylog I documented this so i didn’t forget what worked for me.

5

This is where I had the problem… I did everything else already. This part of the documentation is not clear. All it says is… “Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore and (optionally) javax.net.ssl.trustStorePassword system properties.” but how do I get theDjavax.net.ssl.trustStore variable into the JAVA_OPTS variable?

Well if anyone else reads this, I did get it to work doing the below…

For Centos 7 go to the jvm settings file location which is /etc/sysconfig

open the graylog-server file and edit the file like below…

(# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=[add path to cert (*.jks) here]"

Save file, restart graylog service

That worked for me.

6

hej @martineznet

This part of the documentation is not clear

you are welcome to submit a PR that describe it better https://github.com/Graylog2/documentation/pulls