Node metrics not available (HTTPS behind proxy)

sgarciam · January 24, 2020, 8:21pm

Hi all,

Needing some help to cross the finish line with graylog3. Currently we have a Graylog2 server, running great but its fought the good fight and now graylog3 has entered the building.

As it stands it is sitting behind Apache as a proxy though https. Certificates on apache are wildcard cert/key for our organization. All is well when I browse to the server except the node metrics which cant seem to be able to find a valid cert. Details as follows:

OS: Ubuntu 18.04.3 LTS
Apache: Apache/2.4.29
Graylog: Graylog v3.1.4+1149fe1

Apache ssl conf:

                SSLCertificateFile /etc/apache2/certs/wild.crt
                SSLCertificateKeyFile /etc/apache2/certs/wild.key

                <Location />
                    RequestHeader set X-Graylog-Server-URL "https://my.host.com/"
                    ProxyPass https://127.0.0.1:9000/
                    ProxyPassReverse https://127.0.0.1:9000/
                </Location>

Graylog conf:

http_bind_address = 127.0.0.1:9000
http_enable_tls = true
http_tls_cert_file = /etc/apache2/certs/wild.crt
http_tls_key_file = /etc/apache2/certs/wild.key
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

Cert permissions

ll /etc/apache2/certs/
drw-r-xr-x 5 root root 4096 Jan 24 14:03 ./
drwxr-xr-x 5 root root 4096 Jan 24 09:55 ../
-rw-r-xr-x 1 root root 3783 Jan 24 09:54 wild.crt*
-rw-r-xr-x 1 root root 1958 Jan 24 09:54 wild.key*

The errors I’m getting are one per second once the server is up and running repeating the same messages over and over again:

2020-01-24T15:09:23.340-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-01-24T15:09:25.195-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system/metrics/multiple on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

If I try to click on the node to go into its details I get this lovely trace stack

Exception: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-01-24T15:12:07.197-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system/metrics/multiple on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-01-24T15:12:07.940-05:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) ~[?:1.8.0_232]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[graylog.jar:?]
        at org.graylog2.rest.resources.cluster.ClusterSystemResource.jvm(ClusterSystemResource.java:90) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_232]
        at sun.security.validator.Validator.validate(Validator.java:262) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ~[?:1.8.0_232]
        ... 59 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_232]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_232]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_232]
        at sun.security.validator.Validator.validate(Validator.java:262) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ~[?:1.8.0_232]
        ... 59 more
2020-01-24T15:12:07.965-05:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:1.8.0_232]
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) ~[?:1.8.0_232]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) ~[?:1.8.0_232]
        at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300) ~[graylog.jar:?]
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[graylog.jar:?]
        at org.graylog2.rest.resources.cluster.ClusterSystemPluginResource.list(ClusterSystemPluginResource.java:76) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_232]
        at sun.security.validator.Validator.validate(Validator.java:262) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ~[?:1.8.0_232]
        ... 59 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_232]
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_232]
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:1.8.0_232]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_232]
        at sun.security.validator.Validator.validate(Validator.java:262) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:1.8.0_232]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:1.8.0_232]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ~[?:1.8.0_232]
        ... 59 more
2020-01-24T15:12:08.004-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system/inputstates on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-01-24T15:12:08.339-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2020-01-24T15:12:09.195-05:00 WARN  [ProxiedResource] Unable to call https://127.0.0.1:9000/api/system/metrics/multiple on node <d9369ac5-6c33-4df7-91a8-b68d94624eb5>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Now, I’ve tried making a self signed cert for 127.0.0.1 following: https://docs.graylog.org/en/3.1/pages/configuration/https.html#creating-a-self-signed-private-key-certificate . For this I leave the wildcard certs for apache and change the cert paths on the Graylog config to the self signed ones but I get to the same spot with the same mesages.

Summary, Everything works as expected over https except node metrics… Thoughts?

Regards,
Stephen

jan · January 26, 2020, 5:58pm

So you have apache playing your proxy for the outgoing connection - fine.But you have configured also that you have Graylog terminating https.

The certificate needs to have 127.0.0.1 included to be valid for the localhost, plus http_listen_uri needs to be https://127.0.0.1
This is not done automatically when you set http_enable_tls the default is http.

sgarciam · January 27, 2020, 2:03pm

So I’m overcomplicating the issue then. Good to know. Im leaving apache running as proxy forwarding any http requests to https and graylog itself running internally on http.

Everything seems to be working as it should

As far as best practices for graylog, is that a sound approach?

-S

jan · January 27, 2020, 2:25pm

As far as best practices for graylog, is that a sound approach?

when you have a single server setup this can be done like this. Yes.

I will not give the word for best practice on this kind of setup - but, it has some advantages and some downsides. BUT in a small environment the advantages are more.

system · February 10, 2020, 2:25pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog 3.2 https, one small problem left Graylog Central (peer support)	7	632	March 23, 2020
Apache HTTPS not working Graylog Central (peer support)	24	7107	March 25, 2019
Graylog behind 2 reverse proxies Graylog Central (peer support)	6	738	September 1, 2020
Best aproach for SSL in graylog Graylog Central (peer support)	17	2973	October 2, 2019
PKIX path building failed and no metrics Graylog Central (peer support)	2	431	October 12, 2020

Node metrics not available (HTTPS behind proxy)

Related Topics