Certificate Issue after 3.0 Upgrade

martineznet · July 3, 2019, 6:20pm

I am using SSL to access the web GUI and with my server inputs. I am using the server hostname (not IP) as http_bind_address

When I changed http_bind_address to be an IP, the GUI was no longer accessible, so that is why I had to add make http_bind_address be my server name instead since that is what my certificate has.

My inputs won’t start though. Everything was working before upgrade to 3.0. Previous version was 2.3. Interestingly, all my inputs say ‘NOT RUNNING’ but they are showing current messages being received.

I am seeing the below error…

2019-07-03T14:16:26.370-04:00 WARN [ProxiedResource] Unable to call https://SERVERNAME:9000/api/system/metrics/multiple on node
javax.net.ssl.SSLPeerUnverifiedException: Hostname SERVERNAME not verified:
certificate: sha256/bdxK4kLkJbrgVFqmHTezY3VUz5PGW8iGnX2T/uGo7J4=
DN: EMAILADDRESS=xx@xx.net, CN=SERVERNAME, OU=Technology Dept, O=XX, L=XX, ST=XX, C=XX
subjectAltNames:
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:329) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:282) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:78) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

jan · July 3, 2019, 7:24pm

you should re-read the documentation either in the server.conf or the online documentation about the http_* settings.

martineznet · July 3, 2019, 7:29pm

I did read it… Would you please clarify what you think I am possibly missing?

Under 2.3 I used the server host name (not IP) for SSL connections and everything worked fine… I did not see anything in the notes that would account for why I am seeing the errors above.

When I made http_bind an IP address, then I could no longer access Graylog in my web browser. This is not good.

jan · July 3, 2019, 9:06pm

http://docs.graylog.org/en/3.0/pages/configuration/server.conf.html#web-rest-api

http_bind_address = 127.0.0.1:9000

    The network interface used by the Graylog HTTP interface.
    This network interface must be accessible by all Graylog nodes in the cluster and by all clients using the Graylog web interface.
    If the port is omitted, Graylog will use port 9000 by default.

http_publish_uri = http://$http_bind_address/

    The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all clients using the Graylog web interface.
    The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
    This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address, for example if the machine has multiple network interfaces or is behind a NAT gateway.
    This configuration setting must not be configured to a wildcard address!
    If http_bind_address contains a wildcard IPv4 address (0.0.0.0), http_publish_uri will be filled with the first non-loopback IPv4 address of this machine instead.

http_external_uri = $http_publish_uri

    The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
    The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
    When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
    This setting can be overriden on a per-request basis with the “X-Graylog-Server-URL” HTTP request header.

For your Setup:

http_bind_address = 0.0.0.0:9000

leave all others default and it should work. But with no additional information about the setup that is impossible to say.

martineznet · July 4, 2019, 12:49pm

I tried that. 0.0.0.0:9000… I am still receiving the same certificate error except now it is for the IP address instead of the server name as above…

Basically it’s saying Hostname 10.X.X.X not verified

Earlier when I used the server name as the hostname it was still saying host is not verified though my cert uses server name instead of IP address. Either way, IP or server name, I keep getting the below error since upgrade.

2019-07-04T08:22:06.450-04:00 WARN [ProxiedResource] Unable to call https://10.X.X.X:9000/api/system/metrics/multiple on node
javax.net.ssl.SSLPeerUnverifiedException: Hostname 10.X.X.X not verified:
certificate: sha256/bdxK4kLkJbrgVFqmHTezY3VUz5PGW8iGnX2T/uGo7J4=
DN: EMAILADDRESS=admin@xx.net, CN=SERVERNAME, OU=Technology Dept, O=XX , L=XX, ST=XX, C=US
subjectAltNames:
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:329) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:282) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:78) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

martineznet · July 4, 2019, 5:49pm

This is what I am seeing when I click on an input node within the GUI.

Seems like a JVM cert issue (which was working before) but I don’t know why.

jan · July 5, 2019, 2:56pm

you need to add the CA or the certificate to your JVMs trust store to make that work …

martineznet · July 5, 2019, 6:53pm

Yes I know… I did all that and it was all working when I was on 2.3. I upgraded from 2.3 to 3.0. I see no reason why a simple upgrade should stop my JVM cert from working anymore is baffling to me.

I actually saw another thread that is now closed that spoke about the issue but does not appear to ever been resolved. One person created a SAN but the other person didn’t receive an answer so I know I am not the only person who had experienced their JVM cert no longer being accepted.

jan · July 5, 2019, 6:54pm

did you checked that the update not just removed the jvm keystore from your startup file?

martineznet · July 8, 2019, 11:44am

Not sure which startup file you mean but this is my current cert showing the keystore… This is the one that was working before.

Also Jan are you with the graylog team by chance?

[root@SERVERNAME security]# keytool -keystore cacerts.jks -storepass changeit -list | grep graylog-self-signed-new10 -A1
graylog-self-signed-new10, Jan 9, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 4F:E8:40:25:62:49:A5:7C:44:06:0F:00:B3:92:DF:A5:3E:80:0B:4A

martineznet · July 8, 2019, 2:58pm

Also I did create a SAN and imported it into the JVM and inputs still won’t enable…

2019-07-08T10:55:07.081-04:00 WARN [ProxiedResource] Unable to call https://SERVERNAME:9000/api/system/metrics/multiple on node
javax.net.ssl.SSLPeerUnverifiedException: Hostname SERVERNAME not verified:
certificate: sha256/vGtN82Cc0x30JZ/vXls6Sa7bTmzoPUkSAu1H/MzDbO0=
DN: CN=SERVERNAME, OU=Technology Dept, O=XX, L=XX, ST=XX, C=US
subjectAltNames: [10.X.X.X]
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:329) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:282) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167) ~[graylog.jar:?]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[graylog.jar:?]

jan · July 9, 2019, 6:14am

you need to add the keystore that includes the certificates in the startup file. Like mentioned in

http://docs.graylog.org/en/3.0/pages/secure/sec_graylog_beats.html#custom-jvm-keystore-for-graylog

you need to add this, depending on your file for the JVM settings ( http://docs.graylog.org/en/3.0/pages/configuration/file_location.html )

martineznet · July 9, 2019, 10:36am

I guess what I want to know is was there some significant change between 2.3 and 3.0 that would break a previously working cert that was working fine before in the JVM?

This is not like the first time I am setting this up. It was working fine and then broke. So far I don’t see any reason why a previously working config and cert broke. Unless you know of some change that would cause it to break.

jan · July 9, 2019, 11:43am

You need to adjust the configuration settings from rest_* and web_* to http_* that might result in different hostname/ip usage. As Graylog checks the certificates and connects to itself via the API - so verifying the self used certificate it might not accept its own certificate.

Starting one input is a connection to the API - if that is refused based on bad certificates, the input will not be started.

martineznet · July 9, 2019, 12:32pm

OK, I finally figured it out. I was using SAN but I misconfigured the IP part. I have never had to use a SAN cert before. So here are some of my settings for anyone else reading this in the future to maybe save some time. The below is for a self-signed cert on Centos 7.

Server.conf

http_ bind_address = 0.0.0.0:9000
http_enable_tls = true
http_tls_cert_file = /etc/graylog/certs/cert-file.pem
http_ls_key_file = /etc/graylog/certs/keyfile.pem
http_tls_key_password = YourPassword

SAN Configuration File (graylog.cnf)

[ req ]
default_bits = 2048
distinguished_name = dn
x509_extensions = v3_req
prompt = no
[ dn ]
countryName = US
stateOrProvinceName = XX
localityName = XX
organizationName = XX
organizationalUnitName = Dept
commonName = SERVERNAME
[ v3_req ]
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.X.X.X

I placed mine (graylog.cnf) in /etc/graylog/certs/

Then I created the cert again with the SAN configuration file.

openssl req -x509 -days 3650 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert-san.pem -config graylog.cnf
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:YourPassword

then…

cp cert-san.pem cert-file.pem
cp pkcs8-encrypted.pem keyfile.pem

cd / usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64/lib/security

(cd to whatever your version is of course)

keytool -importcert -keystore cacerts.jks -storepass YourStorePassword -alias graylog-self-signed-san -file /etc/graylog/certs/cert-file.pem

systemctl restart httpd
systemctl restart graylog-server

Check to see if your inputs have started.

jan · July 9, 2019, 12:41pm

glad that you found this.

sneak into this - I had created that because of the various issues that users have. The above mentioned steps are visible in that too.

martineznet · July 9, 2019, 12:58pm

Does it configure SAN too? I did not see that part.

jan · July 9, 2019, 1:13pm

github.com

graylog-labs/shadowCA/blob/master/bin/create_certificate.sh#L165


${SSLBIN} req -new -config ${WDIR}/${CRTNAME}.cnf -key ${WDIR}/${CRTNAME}.key -out ${WDIR}/${CRTNAME}.csr




echo "... prepare sign of ${CRTNAME} request ..."
# create .ext file
# alt_names are added here
cat << EOF > ${WDIR}/${CRTNAME}.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = ${SUBALT}
EOF


echo "... sign of ${CRTNAME} request ..."
# create the certificate
${SSLBIN} x509 -req -in ${WDIR}/${CRTNAME}.csr -CA "${CACERTDIR}"/"${CANAME}".pem -CAkey "${CACERTDIR}"/"${CANAME}".key -CAcreateserial -out ${WDIR}/${CRTNAME}.crt -days "${VALIDDAYS}" -sha256 -extfile ${WDIR}/${CRTNAME}.ext
# create certificate pem
# or fullchain certificate
cat ${WDIR}/${CRTNAME}.crt ${WDIR}/${CRTNAME}.key > ${WDIR}/${CRTNAME}.pem


echo "created ${WDIR}/${CRTNAME} that contains all needed files"

I used it this way because I build in

github.com

graylog-labs/shadowCA/blob/master/bin/create_certificate.sh#L91-L97


# sort array entries and make them uniq
NAMES=($(printf "DNS:%q\n" "${HNAME[@]}" | sort -u))
IPADD=($(printf "IP:%q\n" "${HIP[@]}" | sort -u))


# print each elemet of both arrays with comma seperator
# and create a string from the array content
SUBALT=$(IFS=','; echo "${NAMES[*]},${IPADD[*]}")

one string that contains IP and hostname together.

system · July 23, 2019, 1:14pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[PROBLEM] Ussing HTTPS Graylog Central (peer support)	6	1212	July 14, 2017
Inputs not starting Graylog Central (peer support)	7	3966	March 19, 2019
Unable to start any input after setting up web interface as SSL Graylog Central (peer support)	5	698	June 17, 2019
After SSL - Hostname not verified Graylog Central (peer support)	13	6192	March 24, 2021
Graylog 3.0 Upgrade Problem Graylog Central (peer support)	21	5197	March 22, 2019

Certificate Issue after 3.0 Upgrade

http_bind_address = 127.0.0.1:9000

http_publish_uri = http://$http_bind_address/

http_external_uri = $http_publish_uri

Related Topics