Unable to start inputs after implementing HTTPS on Graylog web interface

[foss4ever]

Issue summary:

I have implemented https/TLS on the Graylog web interface following this guide: Using HTTPS - Configuring Graylog

Now, I’ve got https working on the website, but now my syslog UDP inputs are unable to start - including the new syslog TCP input I just created:

My environment:

Graylog 4.2.4+b643d2b on (Debian 10 on Linux 4.19.0-18-amd64)

Logs:

2022-01-06T08:31:57.929+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/metrics/multiple on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:57.940+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.023+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates/61d5ab8834b9fa6fd275c639 on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.086+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.146+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates/61d5ab8834b9fa6fd275c639 on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.205+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.287+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates/61d5ab8834b9fa6fd275c639 on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.353+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.931+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/metrics/multiple on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:31:59.941+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:32:01.926+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/metrics/multiple on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-01-06T08:32:01.937+01:00 WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Troubleshooting steps taken:

None, really, as I don’t quite know what’s causing these errors.

So… what have I missed, or what might I be doing wrong?

Please do let me know if any important bits of information are missing, and I’ll make sure to update the thread accordingly.

[gsmith]

Hello,

@foss4ever

Unable to call https://graylog.[domain redacted].com:9000/api/system/metrics/multiple on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Where it says “unable to find certification path”. Make sure graylog can access where the certificate are located.

chown graylog:graylog -R /etc/graylog/server/

Next, is this certificate listed as a trusted certificates of your JVM?
Also did you set your Keystore? If not, In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter

-Djavax.net.ssl.trustStore=/path/to/cacerts.jks

Which in your case should be here.

/etc/default/graylog-server

EDIT: I’m not 100% sure but are you using the right certificates for your input?
If you followed that documentation, you should have the following certs.

(graylog-key.pem) and the X.509 certificate (graylog-certificate.pem)

Hope that helps

[foss4ever]

Blockquote
Where it says “unable to find certification path”. Make sure graylog can access where the certificate are located.

Access/permissions should not be an issue:

And, again, the web interface is working with the cert (can access via https).

Blockquote
Next, is this certificate listed as a trusted certificates of your JVM?
Also did you set your Keystore? If not, In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter

I’m not too sure about this java keystore stuff, but here is exactly what I did:

cd /etc/graylog/server
sudo cp /etc/ssl/certs/java/cacerts .

(above path was determined using this article: https://www.baeldung.com/find-java-home)

sudo keytool -importcert -keystore /etc/graylog/server/cacerts -storepass changeit -alias graylog-self-signed -file cert.pem

sudo vi /etc/elasticsearch/jvm.options

Add the following line to the very end of jvm.options config file:

-Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts

Save & exit VI

[gsmith]

Hello,
I have had this issue before which I was able to log into the web interface but my Inputs were not working.

My check list consist of the following.

• Make sure I’m actually using the correct certificates.
• Check Keystore for the correct certificates were installed.
• Make sure I created the proper keystore.
• Permission on the Certificates must be owned by Graylog, if not Graylog will be unable to find certification path.

@foss4ever

WARN [ProxiedResource] Unable to call https://graylog.[domain redacted].com:9000/api/system/inputstates/61d5ab8834b9fa6fd275c639 on node <265afac6-d5af-47ae-b107-7f61973c5a05>: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

So the moral of the story is…

@foss4ever

unable to find valid certification path to requested target

This could be an error on your certificate/s created, OR your using a incorrect one for your input.

Here are a couple post you may want to look at, This helped for understanding what I need to do.

Graylog3 with https (easy tutorial)

My ambient test: -Oracle Linux 7.6 (VMware 15); -Network: NAT; -Graylog3, MongoDB and Elastic: running together; Documentation base for RPM distro. First of all, read Docs » Configuring Graylog » Using HTTPS 0- mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/ 1- openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem 2- openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem 3- openssl pkcs8 -in pk…

WARN [ProxiedResource] Unable to call and unable to start inputs

@pguillermet I’m sorry to hear you still having problems. This did take some time for me to understand how certificates work with graylog-server. One thing was clear, the certs need to be in the right format. Graylog needs to be able to access certs for the input (i.e. permissions). Graylog needs to be able to access the keystore. I can show you what i did but I also know everyone environment is a little bit different, so you might need configure these steps for your needs. The following s…

Hope that helps

[foss4ever]

Ok, I finally managed to get TLS encryption working on the web interface while still being able to start my inputs. Next up is figuring out how to actually get a client to send TLS encrypted syslog input to Graylog.

These are the steps I followed to get things working on my Debian 10 box (thanks to user @dickinsonzach who post this solution: After SSL - Hostname not verified - #12 by dickinsonzach):

Start out by changing directory into /etc/graylog/server

Then do:

sudo vi openssl-graylog.cnf

Insert the following:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = input countrycode
L = input city
O = input organisation name
OU = insert department name
CN = insert graylog FQDN (i.e. graylog.yourdomain.com)

[v3_req]
keyUsage = keyEncipherment, dataEncipherment, nonRepudiation, digitalSignature
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
IP.1 = insert IP-address of graylog server
DNS.1 = insert graylog FQDN

Save & exit VI.

Now create a certificate using the config we just set up:

sudo openssl req -x509 -days 1095 -nodes -newkey rsa:4096 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem

Convert the pkcs5 file into pkcs8 format:
sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

Create an encrypted version of the pkcs8 certificate file:
sudo openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:*insert a cert passphrase*

Import the cert.pem created initially, into the keystore:
sudo keytool -importcert -keystore /etc/ssl/certs/java/cacerts -storepass changeit -alias *insert graylog FQDN* -file cert.pem

Give the graylog user ownership of all the certs:
sudo chown -R graylog:graylog /etc/graylog/

Edit the graylog server config to enable TLS encryption:
sudo vi /etc/graylog/server/server.conf

In section “HTTPS settings”, make changes like so:

http_enable_tls = true
http_tls_cert_file = /etc/graylog/cert.pem
http_tls_key_file = /etc/graylog/pkcs8-encrypted.pem
http_tls_key_password = [insert cert passphrase]

Finally, do:

systemctl restart graylog-server.service

And test that Graylog web interface can now be accessed over https - https://insert graylog FQDN:9000/
Also check that the inputs are still working/running.

[gsmith]

That’s awesome.

@foss4ever

Next up is figuring out how to actually get a client to send TLS encrypted syslog input to Graylog.

I used nxlog for my log shipper. You could either use it a single install ( DL it from the web site) OR use Graylog-Sidecar but the configuration file should be the same.

This is an Example of one client in my lab.

NXLOG_Config

define ROOT C:\Program Files (x86)\nxlog

define CERTDIR %ROOT%\cert  <-- This is where I store my certs on Windows OS

define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log


Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _fileop>
    Module xm_fileop
    # Check the log file size every hour and rotate if larger than 5 MB
    <Schedule>
        Every 1 hour
        <Exec>
            if (file_exists('%LOGFILE%') and file_size('%LOGFILE%') >= 5M)
                file_cycle('%LOGFILE%', 8);
        </Exec>
    </Schedule>
    # Rotate log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>

<Extension gelf>
    Module      xm_gelf
 </Extension>

<Input in>
    Module      im_msvistalog   
</Input>

<Output out>
    Module      om_ssl 
    Host        graylog.domain.com
    Port        51412
    OutputType  GELF_TCP 
    CertFile    %CERTDIR%/graylog3-certificate.pem
    CertKeyFile %CERTDIR%/graylog3-key.pem
    CAFile      %CERTDIR%/cert3.pem
    KeyPass     secret 
    AllowUntrusted  true   
    Exec $Hostname = hostname_fqdn();
    Exec $FullMessage = $raw_event;
    #Exec        to_syslog_snare();
</Output>

<Route >
    Path        in => out
</Route>

EDIT: I should made mention that I was using GELF input TLS/TCP.

I think this is what you wanted.

Encrypted log transfer | NXLog Docs

If you don’t want tot go that route Graylog-Sidecar is a better option. You can find more here

Hope that helps