Email Alerts self-signed cert - unable to find valid certification path to requested target

[erasedhammer]

1. Describe your incident:
I recently upgraded from graylog 4.4 to 5.0, and my email notifications have started failing.
These notifications go out to a locally hosted internal-only postfix mail server. I have not made any changes to the server.conf.

2. Describe your environment:

• OS Information:
  Debian 11 5.10.0-21-amd64

• Package Version:
  graylog-server/stable,now 5.0.5-1 amd64 [installed]

Service logs, configurations, and environment variables:

An error occurred while trying to send an email! (triggered 5 minutes ago)
The Graylog server encountered an error while trying to send an email. This is the detailed error message: org.apache.commons.mail.EmailException: Sending the email to the following server failed : smtp.domain.com:587 (javax.mail.MessagingException: Could not convert socket to TLS; nested exception is: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

# Email transport
transport_email_enabled = true
transport_email_hostname = smtp.domain.com
transport_email_port = 587
transport_email_use_auth = true
transport_email_use_tls = true
transport_email_auth_username = alerts@domain.com
transport_email_auth_password = pass
transport_email_subject_prefix = [Log-Alerts]
transport_email_from_email = alerts@domain.com

3. What steps have you already taken to try and solve the problem?
The common solution I was seeing was to add the self-signed mail certificate to the jvm key store. Unfortunately, this is what I did when I set it all up originally, but I readded it anyways:

keytool -importcert -file /etc/ssl/certs/mail-cert.crt -alias mailtlscert -keystore /etc/ssl/certs/java/cacerts -storepass changeit  
Warning: use -cacerts option to access cacerts keystore
Certificate already exists in keystore under alias <debian:mail-cert.pem>
Do you still want to add it? [no]:  yes
Certificate was added to keystore

I also used -cacert just in case (but it was the same keystore):

keytool -importcert -file /etc/ssl/certs/mail-cert.crt -alias mailtlscert -cacerts -storepass changeit                                      
keytool error: java.lang.Exception: Certificate not imported, alias <mailtlscert> already exists

I have zabbix installed on the same box as graylog, and email notifcations work just fine there.
I have also already restarted graylog and the OS.

4. How can the community help?

Whats wrong here? I looked over the default server.conf for any new options, but dont see anything different.

[gsmith]

Hey @erasedhammer

I have a feeling when this node was updated it probably updated JAVA (i.e, OpenJDK 17 (This is embedded in Graylog 5.0 and does not need to be separately installed.), also and this is why your recieveing “unable to find valid certification path to requested target” . I see you using the default java keystore" cacerts’.

I would execute a global search like this.

root# locate cacerts

This is what I found out.

root@ansible:/etc/graylog/server# locate cacerts
/etc/default/cacerts
/etc/ssl/certs/java/cacerts
/usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts
/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
/usr/share/elasticsearch/jdk/lib/security/cacerts
/usr/share/graylog-server/jvm/lib/security/cacerts
/usr/share/opensearch/jdk/lib/security/cacerts
root@ansible:/etc/graylog/server#

I ended up using /usr/share/graylog-server/jvm/lib/security/cacerts .

[erasedhammer]

I still do have openjdk11 installed, I guess I can go ahead and remove now.

These are all the cacerts:

/usr/share/opensearch/jdk/lib/security/cacerts
/usr/share/graylog-server/jvm/lib/security/cacerts
/usr/share/elasticsearch/jdk/lib/security/cacerts
/usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts
/etc/graylog/cacerts.jks
/etc/ssl/certs/java/cacerts
/etc/default/cacerts

The one I used is:
/etc/graylog/cacerts.jks -> /etc/ssl/certs/java/cacerts

Just added the cert to /usr/share/graylog-server/jvm/lib/security/cacerts and after a graylog restart its working. Thanks!

[gsmith]

Hey @erasedhammer

I just found this out yesterday Glad it helped you out.