After SSL - Hostname not verified

dickinsonzach · March 5, 2021, 2:35pm

I was able to configure SSL with self signed certs

Mongo, Elastic, and Graylog on the same server

All inputs, alerts, etc… are working

BUT…
All my Inputs show NOT RUNNING

In the log I see…
2021-03-05T09:27:28.737-05:00 WARN [ProxiedResource] Unable to call https://…/api/system/inputstates on node <000>: Hostname host not verified:
certificate:

I’m not sure how to verify it.

I’ve tried several versions of the http_publish_uri in server.conf
https://host:9000
https://host.domain.com:9000

I’ve tried several versions of the http_bind_address in server.conf
hostname:9000
hostname.domain.com:9000

Thank you!

dickinsonzach · March 5, 2021, 5:04pm

I’m thinking I messed up the cert some where.

certificate: sha256/
DN: CN= StorageXX, OU=XX, O=XX, L=XX, ST=XX, C=XX
subjectAltNames:

I edited some of that stuff out to obvuscate. But not the subjectAltNames: , that’s how it looks
and if I look at the cert in my browser, I don’t see any mention of my hostname

But when I look at the cert I’m using I see the info.

And something like this seems OK:

keytool -list -v -keystore keystore.jks

I feel like I am very close…

dickinsonzach · March 5, 2021, 6:22pm

Thought I had it with a bad Common Name, but now it looks right and the issue persists:

dickinsonzach · March 5, 2021, 7:05pm

Guess I’ll put this to bed for the week. I thought I had it again with

-Djavax.net.ssl.trustStore= and -Djavax.net.ssl.trustStorePassword=

needing to be set in the graylog-server file

I am using the default cacerts store with a new password

No luck

ttsandrew · March 5, 2021, 7:50pm

What are you using for your SAN names? I’ve configured ours for DNS=fqdn and IP=ip address of the host, and it works for the web interfaces and the inputs.

Subject:
CN = graylog.contoso.com
OU = IT
O = Awesome IT
L = Kansas City
S = Missouri
C = US

Subject Alternative Name:
DNS Name=graylog01
DNS Name=graylog01.contoso.com
IP Address=192.168.1.220

dickinsonzach · March 5, 2021, 9:12pm

I have something that looks like this:

CN=graylog.domain.com
OU=Storage
O=Company
L=GR
ST=MI
C=US
subjectAltNames:

At what step in the instructions (Using HTTPS — Graylog 4.0.0 documentation) do you set the SANs on the cert?

It prompts me for all sorts of info, but I don’t see where I put SANs.

Thank you, Zach.

ttsandrew · March 5, 2021, 9:21pm

In the document you linked it’s in the request file under alt_names (the block under “Create a file named openssl-graylog.cnf with the following content (customized to your needs):”), but it can vary based on the CA/request you’re submitting. We use an AD enterprise CA and our request looks like this:

user@graylog01:/etc/graylog/certs/domain$ cat graylog01-cert-req.cnf
[ req ]
default_bits = 2048
default_keyfile = key.pem
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:graylog01, DNS:graylog01.contoso.com, DNS:192.168.1.220
[ req_distinguished_name ]
0.organizationName = Awesome IT
organizationalUnitName = IT
commonName = graylog01.contoso.com
countryName = US
stateOrProvinceName = Missouri
localityName = Kansas City

gsmith · March 6, 2021, 2:09am

@dickinsonzach
I concur with @ttsandrew about having SAN, and this is a peak at my keystore for my lab certificate. Sorry I had to block my private info but you can see in the red box it shows what @ttsandrew was saying.

Since this was a lab VM I didn’t use a config file, I used my /etc/pki/tls/openssl.cnf file instead. I’m lazy Shown below in red boxes is where i configured my SAN.

dickinsonzach · March 6, 2021, 1:30pm

This is all very helpful. I have tried to detail my process here:

Generate a blank keystore.jks
keytool -genkey -keyalg RSA -validity 1095 -keystore keystore.jks

Create a cert using my req.txt which has my SANs in it
openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config req.txt -keyout pkcs5-plain.pem -out cert.pem

Import the cert into the blank keystore.jks
keytool -importcert -keystore keystore.jks -storepass mypassword -alias host.domain -file cert.pem

Check to see if the keystore alias looks good. It does complete with SANs
keytool -list -v -keystore keystore.jks -alias host.domain

Do some openssl work to get things in the right format
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

openssl pkcs8 -in pkcs5-plain.pem -topk8 -v2 des3 -out pkcs8-encrypted.pem -passout pass:mypassword

Export keystore.jks to .p12 so openssl can work with it
keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12

More SSL work to generate certs in the right format
openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem

openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem

openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

Import keystore.jks into the default cacerts store so I don’t have to mess with Java arguments
keytool -import -trustcacerts -file graylog-certificate.pem -alias host.domain -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts

Check to see if the alias looks correct in the default cacerts. It does complete with SANs
keytool -list -v -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -alias host.domain

Give the graylog user rights to all the files where the install is and I’ve been doing the cert work
chown graylog:graylog -R /etc/graylog

Reboot server and check the log
tail -f /var/log/graylog-server/server.log

/etc/graylog/server/server.conf
http_bind_address = host.domain:9000
http_enable_tls = true
http_tls_cert_file = /etc/graylog/graylog-certificate.pem
http_tls_key_file = /etc/graylog/graylog-key.pem

Everything looks fine in the log. Last message is that all my inputs are running, but the Web UI is un-reachable. Browser says: ERR_SSL_KEY_USAGE_INCOMPATIBLE

dickinsonzach · March 6, 2021, 4:21pm

https://docs.graylog.org/en/4.0/pages/configuration/https.html

The part I’m struggling with is that we generate a cert.pem from a request file, and the next work with it is when we import it into the copied cacerts file

keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem

I see how that will get the cert, including the SANs, into the copied cacerts file.

BUT the whole middle part has us starting with keystore.jks, exporting that as .p12, so we can work with it and eventually end up with the cert and key file to use in the Graylog config.

Where does that keystore.jks file com from and how do we get the SANs/cert from the request in there?

Thank you, Zach,.

gsmith · March 8, 2021, 11:08pm

@dickinsonzach

Sorry for the delay, I had work over the weekend.

Correct me if im wrong, but I believe this issue is related to the value of the ‘KeyUsage’ parameter in the SSL config of ‘v3_req’.
My problem might be a little different but I solved this by changing keyUsage = keyEncipherment, dataEncipherment to keyUsage = nonRepudiation, digitalSignature, keyEncipherment in the section v3_req in file openssl-graylog.cnf. This means you might have to recreate your certs again.

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = Some-State
L = Some-City
O = My Company
OU = My Division
CN = graylog.example.com

[v3_req]
keyUsage = keyEncipherment, dataEncipherment, **nonRepudiation**
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 203.0.113.42
DNS.1 = graylog.example.com

I belive you generate it here:

If you need to add more certs to the keystore you can always use keytool to import it.

keytool -import -trustcacerts -file some_file.pem -alias dns.name.of.server -keystore keystore.jks

dickinsonzach · March 9, 2021, 1:11pm

Thank you to the community with a special shout out to @gsmith

In the end I ended up doing the following. It seems the documentation is a general guide to ALL the ways in which you can secure Graylog with SSL/TLS/Cert. In the end, all I wanted to do was use a self cert in the default Java Keystore.

Create a cert from a config file. I did have to use keyUsage = nonRepudiation, digitalSignature, keyEncipherment as suggested by @gsmith to get it to work with Chrome and not disable TLS 1.3 I made mine valid for 3 years
openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem

Convert the pkcs5 to pkcs8
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

Create an encrypted version of the pkcs8
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:changeit

Make sure my entry was not already in the keystore
keytool -list -v -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -alias host.domain -storepass changeit

It was because of previous work, so delete it
keytool -delete -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -alias host.domain -storepass changeit

Import the cert.pem created above into the keystore
keytool -importcert -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -storepass changeit -alias host.domain -file cert.pem

Verify the new cert is in the keystore
keytool -list -v -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -alias host.domain -storepass changeit

Clean up all my other cert stuff that didn’t work
rm /etc/graylog/.

Copy the new certs from where I had been working to where I want my certs
cp /tmp/zach/. /etc/graylog

Give the graylog user ownership of all the certs. Otherwise you get a cannot verify private key error because Graylog cannot access the files
chown graylog:graylog -R /etc/graylog

Edit the graylog server config
/etc/graylog/server/server.conf
http_enable_tls = true
http_tls_cert_file = /etc/graylog/cert.pem
http_tls_key_file = /etc/graylog/pkcs8-encrypted.pem
http_tls_key_password = changeit

Add the path and password to the default Java Keystore. You don’t have to do this if you don’t change the password. I had, but obfuscate in this post
/etc/sysconfig/graylog-server GRAYLOG_SERVER_JAVA_OPTS=
-Djavax.net.ssl.trustStore=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-3.el8_3.x86_64/jre/lib/security/cacerts -Djavax.net.ssl.trustStorePassword=changeit

Restart the Graylog server
systemctl restart graylog-server

Watch the logs
tail -f /var/log/graylog-server/server.log

At this point SSL on the website, all APIs, TLS 1.3, and Chrome are working. Now to see I can get an Input to use it See you back in the forums, Zach.

gsmith · March 10, 2021, 12:02am

@dickinsonzach
Nice, seams like your getting closer.
FYI:
Be careful when updating java on your server, that path might change on you.

system · March 24, 2021, 12:03am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issues with reverse DNS and picky certs Graylog Central (peer support)	2	1058	August 21, 2018
SSL Issues ProxiedResource Unable to call Hostname not verified Graylog Central (peer support)	5	2488	December 15, 2021
Java - certification issue Graylog Central (peer support)	4	1146	June 14, 2017
SSL Certs - Lesson Learned Graylog Central (peer support)	2	3906	March 27, 2018
Graylog2 Using https Graylog Central (peer support)	10	8051	November 29, 2017

After SSL - Hostname not verified

Related Topics