Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
I am setting up graylog for the first time and during preflight I continue to get the error:
Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.;Unable to retrieve version from indexer node.
In the mongodb log file I get no errors.
In the datandoe log file I get:
Caught exception while handling client http traffic, closing connection Netty4HttpChannel
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
In the server log file I get:
ERROR [VersionProbe] Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.
INFO [VersionProbe] Indexer is not available. Retry #1/1
ERROR [VersionProbe] Unable to retrieve version from indexer node:
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 1 attempts.
When I search deeper by checking the status with curl I get:
ALPN, offering h2
- ALPN, offering http/1.1
- CAfile: /etc/pki/tls/certs/ca-bundle.crt
- TLSv1.0 (OUT), TLS header, Certificate Status (22):
- TLSv1.3 (OUT), TLS handshake, Client hello (1):
- TLSv1.2 (IN), TLS header, Certificate Status (22):
- TLSv1.3 (IN), TLS handshake, Server hello (2):
- TLSv1.2 (IN), TLS header, Finished (20):
- TLSv1.2 (IN), TLS header, Unknown (23):
- TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
- TLSv1.3 (IN), TLS handshake, Certificate (11):
- TLSv1.2 (OUT), TLS header, Unknown (21):
- TLSv1.3 (OUT), TLS alert, unknown CA (560):
- SSL certificate problem: self-signed certificate in certificate chain
- Closing connection 0
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
I followed this direct deployment, single node RHEL installation guide
2. Describe your environment:
-
OS Information:
Red Hat Enterprise Linux release 9.6 (Plow)
-
Package Version:
(Open Version Type)
graylog-datanode-6.2.4-1.x86_64
graylog-6.3-repository-1-1.noarch
graylog-server-6.3.1-1.x86_64 -
Service logs, configurations, and environment variables:
/etc/mongod.conf -------------------------------------------------------------------------------------------
#mongod.conf
#where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
#Where and how to store data.
storage:
dbPath: /var/lib/mongo
#how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
#network interfaces
net:
port: 27017
bindIp: X.X.X.X, 127.0.0.1
#security:
#operationProfiling:
#replication:
#sharding:
##Enterprise-Only Options
#auditLog:/etc/graylog/datanode/datanode.conf ------------------------------------------------------------------
node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = omitted
root_password_sha2 =
mongodb_uri = mongodb://127.0.0.1/graylog
bind_address = X.X.X.X
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 6g/etc/sysconfig/graylog-server --------------------------------------------------------------------------------
#Path to a custom java executable. By default the java executable of the
#bundled JVM is used.
#JAVA=/usr/bin/java
#Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS=“-Xms6g -Xmx6g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks -Djavax.net.ssl.trustStorePassword=changeit”
#Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true”
#Fix for log4j CVE-2021-44228
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true”
#Pass some extra args to graylog-server. (i.e. “-d” to enable debug mode)
GRAYLOG_SERVER_ARGS=“”
#Program that will be used to wrap the graylog-server command. Useful to
#support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=“”/etc/graylog/server/server.conf ---------------------------------------------------------------------------
setup = false
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = omitted
root_password_sha2 = omitted
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = X.X.X.X:9000
http_publish_uri = https://[hostname]:9000/
http_external_uri = https://[hostname]:9000/
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/GraylogChained-ordered.pem
http_tls_key_file = /etc/ssl/private/graylogPK.key
stream_aware_field_types=false
elasticsearch_tls_cert_file = /etc/ssl/certs/GraylogChained-ordered.pem
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 12h
message_journal_max_size = 5gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://[hostname]:27017/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts
enable_preflight_web = true //Added this line because gui had stopped working3. What steps have you already taken to try and solve the problem?
I tried to upload the certificate it didn’t work.
I followed this guide to manually add it:
https://graylog.org/post/how-to-guide-securing-graylog-with-tls/I imported to the java keystore the chained certficate PEM and the CA crt, updated the config files to reflect the import, converted to the correct .pem format, put the chained cert in the correct order, I followed the documentation permission grants for the files also the required open ports documentation and restart the server after changes.
I also attempted adding the net configuartion in mongodb but that didn’t work:
#net:
#tls:
#mode:requireTLS
#certificateKeyFile:/etc/ssl/certs/mongoDBCertKeyFile.pem
#security:
#clusterAuthMode: x509I also then attempted by importing the certificates to the opensearch keystore and editing the yml file but that didn’t work
I checked to make sure the path to the keystore was pointing to the correct keystore and it is
I checked the keystore contents to make sure it was not empty and the cert actually imported.Since it’s an rhel setup I also:
sudo cp /etc/ssl/certs/GraylogChained-ordered.pem /etc/pki/ca-trust/source/anchors
Update-ca-trustI also added the VMs host name to the /etc/hosts file
4. How can the community help?
I have followed all the guides and documentation yet I can’t figure out the problem. I would appreciate some guidance and advice on what I missed or did wrong please, thank you!