Update: Got it working… mostly.
I updated to 3.1 in the hopes that might help. (it actually made troubleshooting more difficult as graylog does not start and stop gracefully any longer with https configurations in server.conf that it doesn’t like)
I went through the instructions on creating self signed certificate and then importing to a fresh copy of cacerts.jks one more time from a blank slate while on 3.1 making sure every t was crossed and i dotted. Still didn’t work. I then took Tobias’ instructions of trying without an encrypted key and that worked. So here is the final rundown of what worked for me (from https://docs.graylog.org/en/3.1/pages/configuration/https.html#ssl-setup)
Create Self signed private key/certificate
openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret
Adding a self signed certificate to the JVM trust store
(on Ubuntu 18.04.2 LTS)
cp -a /etc/ssl/certs/java/cacerts /etc/ssl/certs/graylog/cacerts.jks
keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem
Then in my server.conf I have:
http_publish_uri = https://graylog.domain.com:9000/
http_enable_tls =true
http_tls_cert_file =/etc/ssl/certs/graylog/cert.pem
http_tls_key_file =/etc/ssl/certs/graylog/pkcs8-plain.pem
#http_tls_key_password =secret
And in my /etc/default/graylog-server:
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/cacerts.jks"
So that works for me. I just wish I could figure out why the tls key password isn’t working. I have it chmod’d just like the certificate, I even changed the owner to graylog.