Graylog 3.3.9 enabling TLS, inputs no longer starting

wterstegen · December 18, 2020, 1:53pm

I have already looked at the forums and see some more topics describing my same issues, but so far I cannot see what would be wrong in my situation.

error looks to indicate issue with my certificate in the Java keystore But with keytool I can see the certificate is imported.

This concerns a Graylog setup with docker and docker-compose.

We use Active Directory Certificate Services and csr has been send to ADCS for certificate generation.

key has been generated with openssl
csr created with openssl
certificate requested via ADCS with SAN attributes for IPAddress and DNS. (DNS attribute with both IP and DNS name)
Webserver template used is specific one created for Graylog with Server and Client authentication, per documenation
our company root and intermediate CA have been concatenated in the certificate

Keytool lookup displays below information on my certificate

#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]
#7: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]
#8: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
IPAddress: 10.31.252.35
DNSName: 10.31.252.35
DNSName: graylog-nl.ourdomain.com

Java keystore has been copied and referenced in docker-compose file. Here my complete docker-compose file:

services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:3
    volumes:
      - mongo_data:/data/db
    networks:
      - graylog
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.10
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:3.3
    volumes:
      - graylog_data:/usr/share/graylog/data
      - /opt/graylog/plugin/graylog-plugin-integrations-3.3.9.jar:/usr/share/graylog/plugin/graylog-plugin-integrations-3.3.9.jar
      - /opt/graylog/plugin/graylog-plugin-enterprise-integrations-3.3.9.jar:/usr/share/graylog/plugin/graylog-plugin-enterprise-integrations-3.3.9.jar
      - /opt/graylog/plugin/graylog-plugin-enterprise-3.3.9.jar:/usr/share/graylog/plugin/graylog-plugin-enterprise-3.3.9.jar
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=<secret omitted>
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=<omitted>
      - GRAYLOG_HTTP_EXTERNAL_URI=https://graylog-nl.domain.com:9000/
      - GRAYLOG_HTTP_PUBLISH_URI=https://graylog-nl.domain.com:9000/
      - GRAYLOG_JAVA_OPTS=-Djavax.net.ssl.trustStore=/usr/share/graylog/data/ssl/cacerts
      - GRAYLOG_HTTP_ENABLE_TLS=true
      - GRAYLOG_HTTP_TLS_CERT_FILE=/usr/share/graylog/data/ssl/graylog-nl-san_ca-chain.cer
      - GRAYLOG_HTTP_TLS_KEY_FILE=/usr/share/graylog/data/ssl/pkcs8-plain.pem
    networks:
      - graylog
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # Syslog TCP Secure
      - 8514:8514/tcp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
      # raw/tcp
      - 5555:5555/tcp
networks:
  graylog:
    driver: bridge
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_data:
    driver: local

Here the logging from docker logs command:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:1.8.0_275]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_275]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_275]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) ~[?:1.8.0_275]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) ~[?:1.8.0_275]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:190) ~[graylog.jar:?]
        at org.graylog2.rest.resources.cluster.ClusterSystemPluginResource.list(ClusterSystemPluginResource.java:76) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_275]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:1.8.0_275]
        ... 62 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_275]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_275]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_275]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:1.8.0_275]
        ... 62 more
2020-12-18 13:03:48,105 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:1.8.0_275]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) ~[?:1.8.0_275]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:1.8.0_275]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) ~[?:1.8.0_275]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) ~[?:1.8.0_275]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) ~[?:1.8.0_275]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) ~[?:1.8.0_275]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:190) ~[graylog.jar:?]
        at org.graylog2.rest.resources.cluster.ClusterSystemResource.jvm(ClusterSystemResource.java:91) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_275]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_275]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:1.8.0_275]
        ... 62 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_275]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_275]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451) ~[?:1.8.0_275]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323) ~[?:1.8.0_275]
        at sun.security.validator.Validator.validate(Validator.java:271) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223) ~[?:1.8.0_275]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:1.8.0_275]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:1.8.0_275]
        ... 62 more
2020-12-18 13:03:48,124 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:49,860 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system/metrics/multiple on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:52,767 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system/metrics/multiple on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:52,792 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:54,767 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system/metrics/multiple on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:56,765 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system/metrics/multiple on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-12-18 13:03:57,775 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog-nl.domain.com:9000/api/system on node <21471aec-61c1-44c8-846f-f6e4bd83b392>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Also in the GUI accessing the logs will also result in an error referring me to the graylog logs

I hope someone can see the error in my setup. Disabling TLS, will restore functionality of the inputs and accessing the GUI logging.

Thanks,

Wim

wterstegen · December 22, 2020, 9:35am

Some actions I tried in the meantime without success unfortunately:

included the entry for GRAYLOG_JAVA_OPTS in quotes. As this is done in the docker-compose examples also for ElastiSearch
removed the concatenated certificate in the Java keystore and imported each certificate separately, so certificate, intermediate root CA and root CA.

All still with same result. For the WebGUI HTTPS is working fine except communications using Java keystore

I tried some more things, and from what it now looks is that my GRAYLOG_JAVA_OPTS is not being read and used. Java is still using the original keystore in the container located in /usr/local/openjdk-8/lib/security/

If I import my concatenated certificate in this original keystore, everything works.

I also included -Djavax.net.ssl.trustStorePassword= with default password in case it was an issue with opening the keystore even with default password, but also without success.

Also permisisons within container look to be fine, also running keytool from within container can access the Java keystore

graylog@fc3ce0cb49a6:~$ ls -l /usr/share/graylog/data/ssl/
total 332
-rw-r--r--. 1 root    root      1270 Dec 22 06:13 TP-RootCA.cer
-rw-r--r--. 1 root    root      1670 Dec 22 06:13 TP-SubordinateC3.cer
-r--r--r--. 1 graylog graylog 149491 Dec 22 07:42 cacerts
-r--r--r--. 1 graylog graylog 147467 Nov 18 00:19 cacerts-original.bak
-rwxr-xr-x. 1 root    root      2780 Dec 22 06:04 graylog-nl-san.cer
-rwxrwxrwx. 1 root    root      5720 Dec 16 14:13 graylog-nl-san_ca-chain.cer
-rwxrwxrwx. 1 root    root      5690 Dec  1 16:10 graylog-nl_ca-chain.cer
-rwxrwxrwx. 1 root    root      3488 Dec  1 16:11 pkcs8-encrypted.pem
-rwxrwxrwx. 1 root    root      3320 Dec 16 13:31 pkcs8-plain.pem

system · January 5, 2021, 9:36am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Odd thing happen after enable https Graylog Central (peer support)	9	1521	November 6, 2019
Input failed to start after https Graylog Central (peer support)	13	1366	March 26, 2020
Java Keystore and Docker Graylog Central (peer support)	2	1496	May 27, 2021
Import internal certificate to java keystore in graylog container Graylog Central (peer support) docker	5	2015	April 18, 2022
Graylog in HTTPS mode with Docker-Compose Graylog Central (peer support)	3	3042	February 27, 2020

Graylog 3.3.9 enabling TLS, inputs no longer starting

Related topics