Graylog & Datanodes break with custom SSL certificates

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
When I turn on SSL for my frontend servers, sing certs signed & trusted from an external CA, I get one of two errors in my server.conf
If i leave http_publish_uri to http these errors occur:
2025-03-10T11:48:56.149Z WARN [ProxiedResource] Failed to call API on node <4b608ba4-ddba-4693-9467-37c80fd9166b>, cause: unexpected end of stream on hxxp://graylog.rng.reddenorthgate.com:9000/… (duration: 2 ms)
2025-03-10T11:48:56.149Z WARN [ProxiedResource] Failed to call API on node <66645273-1487-435a-a92b-a5e5ce28b7f2>, cause: unexpected end of stream on hxxp://winlog.rng.reddenorthgate.com:9000/… (duration: 2 ms)

If i update the http_publish_uri to https://{frontend server DNS name} then the log instead fills up with trust errors:
2025-03-10T11:54:04.138Z WARN [ProxiedResource] Failed to call API on node <4b608ba4-ddba-4693-9467-37c80fd9166b>, cause: None of the TrustManagers trust this certificate chain. (duration: 9 ms)

The frontend graylog servers are running sucessfully with HTTPS at this point, but the backend datanodes break, and i can’t seem to find any documentation about how to correctly assign the Root CA certs to a relevant keystore.

If i work through the preflight and get self-signed certs assigned to the datanodes it works fine, but there is no SSL on the front end.

2. Describe your environment:
2 graylog nodes, 2 graylog-datanode nodes.

  • OS Information:
    RHEL 9.5

  • Package Version:
    graylog-server.x86_64 - 6.1.8-1
    graylog-datanode.x86_64 - 6.1.8-1

  • Service logs, configurations, and environment variables:
    as above.

3. What steps have you already taken to try and solve the problem?

  1. creating a custom jks keystore for graylog-datanode containing the graylog server root CA cert and editing /etc/graylog/datastore/jvm.options to use this keystore, then restarting. no change.
  2. adding the root CA cert to /usr/share/graylog-datanode/jvm/lib/security/cacerts then restarting. no change to errors.
  3. dug through logs, found some extra keystores mentioned:
    /opt/graylog-datanode/opensearch/config/opensearch/opensearch.keystore - Can’t edit (invalid version?)
    /opt/graylog-datanode/opensearch/config/opensearch/http-keystore.p12 - just cointains local node certs.
    /opt/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12 - jsut contains local node certs.
    /opt/graylog-datanode/opensearch/config/keystore.jks - just contains local node certs

also tried starting graylog-datnode with some custom certs:

  • built cert and then built pkcs12/pfx/p12 bundle and added to config. this also failed.

4. How can the community help?

Is there any docs around about how to correctly setup the datanodes to work with frontend server having official certificates. OR are there docs on setting datanode/opensearch up to use certs from an external CA. I’ve been all over the internet for the last few days and google-fu has failed me.

Thank you in advance.

2

Ok so i’m an idiot. and writing out the problem ignited a spark which lead me to a solution.

the issue isn’t with the datanodes (though there still isn’t much ssl documentation on them but i’ll live) bit with the trust chain for the SSL certs as I added them to the frontend graylog servers.

Somehow the Windows root CA cert that i’d imported to all my keystores didn’t work, or behave as expected to create the relevant trust chain.

I imported each server’s PEM format cert into the other server’s graylog keystore and it all worked. whichis fustrating as i thought I knew how to make x509 certs work.

ho hum. ill post back once i’ve figured out why

2 Likes