Strange certificate issue (Caused by: java.security.KeyStoreException: Key protection algorithm not found: java.security.KeyStoreException: Certificate chain is not valid)

vladx · February 18, 2020, 2:51pm

Hi,

At one of my customers I have to use a wildcard certificate for Graylog (*.company.hu). When I use this cert/key I’m getting this error

Caused by: java.security.KeyStoreException: Key protection algorithm not found: java.security.KeyStoreException: Certificate chain is not valid

Any idea how to fix this? Similar set-up with explicit certificate works fine

Thanks
Laszlo

2020-02-18T14:00:57.429+01:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], OutputSetupService [TERMINATED], LookupTableService [TERMINATED], EtagService [TERMINATED], ConfigurationEtagService [TERMINATED], PeriodicalsService [TERMINATED], JournalReader [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED]], FAILED=[JerseyService [FAILED]]}

2020-02-18T14:00:57.430+01:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:

java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}

    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]

    at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]

    at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]

    at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:148) ~[graylog.jar:?]

    at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]

    at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]

    Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]

    Caused by: java.security.KeyStoreException: Key protection  algorithm not found: java.security.KeyStoreException: Certificate chain is not valid

            at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:704) ~[?:?]

            at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]

            at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]

            at java.security.KeyStore.setKeyEntry(KeyStore.java:1174) ~[?:?]

            at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:125) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]

            at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]

            at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]

            at java.lang.Thread.run(Thread.java:834) ~[?:?]

    Caused by: java.security.KeyStoreException: Certificate chain is not valid

            at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:651) ~[?:?]

            at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]

            at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]

            at java.security.KeyStore.setKeyEntry(KeyStore.java:1174) ~[?:?]

            at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:125) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]

            at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]

            at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]

            at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]

            at java.lang.Thread.run(Thread.java:834) ~[?:?]

shoothub · February 18, 2020, 3:15pm

Check, that your keystore also contains intermediate certificate, because wildcard certificate are usually signed by intermediate certificate, not only root certificate.

vladx · February 19, 2020, 8:46am

Hi,

So, I’ve imported all the certs one by one

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 3 entries

, Feb 19, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): 9A:7A:3C:A4:76:CE:AA:EB:45:D7:51:EA:66:B4:3D:0C:80:9E:8A:2D:CA:93:FE:B2:D6:CC:DE:A7:7D:82:FF:04
e-szigno-root, Feb 19, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): 3C:5F:81:FE:A5:FA:B8:2C:64:BF:A2:EA:EC:AF:CD:E8:E0:77:FC:86:20:A7:CA:E5:37:16:3D:F3:6E:DB:F3:78
e-szigno-ssl-ca, Feb 19, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): EA:C2:41:C0:44:0A:36:83:01:11:38:33:36:BC:20:CA:C7:40:9C:20:F6:E8:8D:4F:84:F4:82:7B:E9:19:E3:38

I also instructed graylog to use this certstore

GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks -Djavax.net.ssl.trustStorePassword=xxxxxxxxxxx"

But I still have the same error.

Also in my another system I was able to skip this whole step by adding certs to java keystore, but that system based on CentOS7, and this one is Debian10, so maybe this is why.

Thanks
Laszlo

shoothub · February 19, 2020, 9:33am

Hi,
do you use valid cert from commercial certification authority or your own CA?

If you use wildcard cert from commercial CA, I think, that root cert is already in java truststore. You only need to setup path for certificate and key in graylog server.conf:
https://docs.graylog.org/en/3.2/pages/configuration/https.html

http_enable_tls = true
http_tls_cert_file = /path/to/graylog-certificate.pem
http_tls_key_file = /path/to/graylog-key.pem

If you have intermediate cert, medge it to graylog-certificate.pem with certificate:

cat cert.pem intermediate.pem > graylog-certificate.pem

If you use own CA, you need to insert only root CA to java trustStore:
https://docs.graylog.org/en/3.2/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

vladx · February 19, 2020, 9:48am

Hi,

After following the official documentation, I was able to start graylog and I was able log-in for a few minutes, but after that I’m getting timeouts while Graylog trying to call /api Any idea how to fix this last piece?

Thanks
Laszlo

shoothub · February 19, 2020, 10:02am

Check your server.conf parameters:
http_publish_uri =
http_external_uri =

It should contain https://

https://docs.graylog.org/en/3.2/pages/configuration/server.conf.html#web-rest-api

vladx · February 19, 2020, 5:19pm

Yes, I have these

web_endpoint_uri = https://servername:9000/api/
rest_transport_uri = https://servername:9000/
web_listen_uri = https://servername:9000/
http_publish_uri = https://servername:9000/
http_external_uri = https://servername:9000/

but, I still have

We are experiencing problems connecting to the Graylog server running on https://servername:9000/api/. Please verify that the server is healthy and working correctly

I still may need some last piece of config to make i working… the strange thing is, there are nothing in the logs, the connection simply timeouts

Any idea?

Thanks
Laszlo

jan · February 20, 2020, 11:01am

the question @vladx

what Graylog version did you use?

You use settings that are pre and post Graylog 3.0 …

vladx · February 20, 2020, 1:31pm

I have the latest version on Debian10. Same version with similar config running seamlessly on CentOS 7 with explicit certificate, so I’m thinking it is maybe related to the certificate or to the OS

Here is my version
graylog-3.2-repository/stable,now 1-1 all [installed]
graylog-integrations-plugins/stable,now 3.2.1-1 all [installed]
graylog-server/stable,now 3.2.1-1 all [installed]

Thanks
Laszlo

vladx · February 20, 2020, 2:02pm

I also noticed a major difference between centos7 and debian 10. On debian I have a much newer openssl version
OpenSSL 1.1.1d 10 Sep 2019

Can it cause issues like this? I’ve created a self signed certificate using the official doc, and I have the same symptoms

Thanks
Laszlo

vladx · February 20, 2020, 7:28pm

So, here is the solution

I’ve tried with both commercial and self signed certificates, but all the same. After a few minutes TLS hung, so I changed Graylog back to default and listening only on localhost/http, installed nginx and configured as reverse proxy with TLS and magically it is working fine. I’m almost sure there is some conflict between latest Graylog TLS implementation and Debain 10 or openssl 1.1

What do you think?

Laszlo

jan · February 21, 2020, 8:12am

What do you think?

maybe - that is something that could happen. Can you reproduce that in a VM? Can you open a bug report for that over at github with steps how to reproduce this?

thx
Jan

vladx · February 24, 2020, 3:31pm

I’ll do my best, but first I have to finish some new implementation At least it is working

system · March 9, 2020, 3:31pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.