3.2.2 Message Input Bug

[mschroeder (Michael)]

So I enabled HTTPS and that’s where this “bug” began appearing. Looking at the server.log file I see it’s being pegged with:

2020-03-05T13:55:41.018-05:00 WARN [ProxiedResource] Unable to call https://143.15.6.98:8443/api/system/inputstates on node <51707e26-34de-4563-990d-1008916b3609>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

One after the other these are piling in. So I followed the instructions for using a self signed cert (the one I was using was signed by my companies internal CA which is not trusted in the cacerts java keystore) so I do that and add that cert into the cacerts keystore following all documentation found here: http://docs.graylog.org/en/2.3/pages/configuration/https.html#creating-a-self-signed-private-key-certificate

Even after following those instructions and rebooting the entire box the error still persists. BUT I’m still getting log messages in, even though the inputs say not running. I’m at a loss, but every things working just that log file is going to be massive if it keeps thinking the inputs aren’t running.

[EDIT]

I also see this error in the same log file:

2020-03-05T13:54:54.842-05:00 WARN [ProxiedResource] Unable to call https://143.15.6.98:8443/api/system/metrics/multiple on node <51707e26-34de-4563-990d-1008916b3609>: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The input one only appears when you visit the input tab on the graylog server, but log messages still come in…

[EDIT 2]

startup file for graylog:

[Unit]
Description=Graylog server
Documentation=http://docs.graylog.org/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=on-failure
RestartSec=10
User=graylog
Group=graylog
LimitNOFILE=64000

ExecStart=/usr/share/graylog-server/bin/graylog-server

When a JVM receives a SIGTERM signal it exits with 143.

SuccessExitStatus=143

JAVA_OPTS=javax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts.jks

Make sure stderr/stdout is captured in the systemd journal.

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

[jan (Jan Doberstein)]

he @mschroeder

it does look more like an https enable issue and not a message input bug - or?

Please make your message more clear on this. The error indicate that Graylog is not able to verify the certificate.

If you have an internal CA - why not add the CA certificate to the Java Keystore?

https://docs.graylog.org/en/3.2/pages/secure/sec_graylog_beats.html#adding-of-der-to-jvm-keystore

[mschroeder (Michael)]

Right, that makes more sense for the title and I would agree with you that it’s not accepting the certificate but I placed it in the keystore:

[root@hpctest-graylog ~]# keytool -keystore /etc/pki/ca-trust/extracted/java/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1
graylog-self-signed, Mar 5, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): 58:FF:37:A9:85:DA:B5:5E:EA:70:45:FB:C8:50:65:05:74:43:B4:20

Also, I did place my internal CA signed certificate in the keystore as well but was experiencing the same issue. I read here that it needs to have the IP address of the server in order for everything to work. It’s almost like the startup file isn’t using the right keystore, even though I specify the correct one in my startup script.

[EDIT]

Also, I don’t understand how the inputs can say “not running” in the web interface while I still receive messages from those inputs.

[mschroeder (Michael)]

I found this

mainly adding this:

Add “-Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts.jks” in /etc/sysconfig/graylog-server

and restarting graylog worked for me. I didn’t see this anywhere in the official documentation. Maybe it should be in there.

[Edit]

My mistake, it is here