Initial Installation Preflight Graylog Error: Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.;Unable to retrieve version from indexer node

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I am setting up graylog for the first time and during preflight I continue to get the error:

Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.;Unable to retrieve version from indexer node.

In the mongodb log file I get no errors.

In the datandoe log file I get:
Caught exception while handling client http traffic, closing connection Netty4HttpChannel
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

In the server log file I get:
ERROR [VersionProbe] Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.
INFO [VersionProbe] Indexer is not available. Retry #1/1
ERROR [VersionProbe] Unable to retrieve version from indexer node:
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 1 attempts.

When I search deeper by checking the status with curl I get:
ALPN, offering h2

  • ALPN, offering http/1.1
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
  • TLSv1.0 (OUT), TLS header, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS header, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS header, Finished (20):
  • TLSv1.2 (IN), TLS header, Unknown (23):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS header, Unknown (21):
  • TLSv1.3 (OUT), TLS alert, unknown CA (560):
  • SSL certificate problem: self-signed certificate in certificate chain
  • Closing connection 0
    curl: (60) SSL certificate problem: self-signed certificate in certificate chain

I followed this direct deployment, single node RHEL installation guide

2. Describe your environment:

  • OS Information:

    Red Hat Enterprise Linux release 9.6 (Plow)

  • Package Version:
    (Open Version Type)
    graylog-datanode-6.2.4-1.x86_64
    graylog-6.3-repository-1-1.noarch
    graylog-server-6.3.1-1.x86_64

  • Service logs, configurations, and environment variables:

    /etc/mongod.conf -------------------------------------------------------------------------------------------
    #mongod.conf
    #where to write logging data.
    systemLog:
    destination: file
    logAppend: true
    path: /var/log/mongodb/mongod.log
    #Where and how to store data.
    storage:
    dbPath: /var/lib/mongo
    #how the process runs
    processManagement:
    timeZoneInfo: /usr/share/zoneinfo
    #network interfaces
    net:
    port: 27017
    bindIp: X.X.X.X, 127.0.0.1
    #security:
    #operationProfiling:
    #replication:
    #sharding:
    ##Enterprise-Only Options
    #auditLog:

    /etc/graylog/datanode/datanode.conf ------------------------------------------------------------------
    node_id_file = /etc/graylog/datanode/node-id
    config_location = /etc/graylog/datanode
    password_secret = omitted
    root_password_sha2 =
    mongodb_uri = mongodb://127.0.0.1/graylog
    bind_address = X.X.X.X
    opensearch_location = /usr/share/graylog-datanode/dist
    opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
    opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
    opensearch_logs_location = /var/log/graylog-datanode/opensearch
    opensearch_heap = 6g

    /etc/sysconfig/graylog-server --------------------------------------------------------------------------------
    #Path to a custom java executable. By default the java executable of the
    #bundled JVM is used.
    #JAVA=/usr/bin/java
    #Default Java options for heap and garbage collection.
    GRAYLOG_SERVER_JAVA_OPTS=“-Xms6g -Xmx6g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks -Djavax.net.ssl.trustStorePassword=changeit”
    #Avoid endless loop with some TLSv1.3 implementations.
    GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true”
    #Fix for log4j CVE-2021-44228
    GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true”
    #Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)
    GRAYLOG_SERVER_ARGS=“”
    #Program that will be used to wrap the graylog-server command. Useful to
    #support programs like authbind.
    GRAYLOG_COMMAND_WRAPPER=“”

    /etc/graylog/server/server.conf ---------------------------------------------------------------------------
    setup = false
    is_leader = true
    node_id_file = /etc/graylog/server/node-id
    password_secret = omitted
    root_password_sha2 = omitted
    bin_dir = /usr/share/graylog-server/bin
    data_dir = /var/lib/graylog-server
    plugin_dir = /usr/share/graylog-server/plugin
    http_bind_address = X.X.X.X:9000
    http_publish_uri = https://[hostname]:9000/
    http_external_uri = https://[hostname]:9000/
    http_enable_tls = true
    http_tls_cert_file = /etc/ssl/certs/GraylogChained-ordered.pem
    http_tls_key_file = /etc/ssl/private/graylogPK.key
    stream_aware_field_types=false
    elasticsearch_tls_cert_file = /etc/ssl/certs/GraylogChained-ordered.pem
    disabled_retention_strategies = none,close
    allow_leading_wildcard_searches = false
    allow_highlighting = false
    field_value_suggestion_mode = on
    output_batch_size = 500
    output_flush_interval = 1
    output_fault_count_threshold = 5
    output_fault_penalty_seconds = 30
    processor_wait_strategy = blocking
    ring_size = 65536
    inputbuffer_ring_size = 65536
    inputbuffer_wait_strategy = blocking
    message_journal_enabled = true
    message_journal_dir = /var/lib/graylog-server/journal
    message_journal_max_age = 12h
    message_journal_max_size = 5gb
    lb_recognition_period_seconds = 3
    mongodb_uri = mongodb://[hostname]:27017/graylog
    mongodb_max_connections = 1000
    integrations_scripts_dir = /usr/share/graylog-server/scripts
    enable_preflight_web = true //Added this line because gui had stopped working

    3. What steps have you already taken to try and solve the problem?
    I tried to upload the certificate it didn’t work.
    I followed this guide to manually add it:
    https://graylog.org/post/how-to-guide-securing-graylog-with-tls/

    I imported to the java keystore the chained certficate PEM and the CA crt, updated the config files to reflect the import, converted to the correct .pem format, put the chained cert in the correct order, I followed the documentation permission grants for the files also the required open ports documentation and restart the server after changes.

    I also attempted adding the net configuartion in mongodb but that didn’t work:

    #net:
    #tls:
    #mode:requireTLS
    #certificateKeyFile:/etc/ssl/certs/mongoDBCertKeyFile.pem
    #security:
    #clusterAuthMode: x509

    I also then attempted by importing the certificates to the opensearch keystore and editing the yml file but that didn’t work

    I checked to make sure the path to the keystore was pointing to the correct keystore and it is
    I checked the keystore contents to make sure it was not empty and the cert actually imported.

    Since it’s an rhel setup I also:
    sudo cp /etc/ssl/certs/GraylogChained-ordered.pem /etc/pki/ca-trust/source/anchors
    Update-ca-trust

    I also added the VMs host name to the /etc/hosts file

4. How can the community help?

I have followed all the guides and documentation yet I can’t figure out the problem. I would appreciate some guidance and advice on what I missed or did wrong please, thank you!

Hi @log-gray

welcome to the community.

Could you please remove the elasticsearch_tls_cert_file certificate from your graylog configuration and restart preflight? With datanode tls between graylog and opensearch is handled automatically

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.