Unable to access GUI after datanode certificate expire

[ajd]

Hello everyone,

1. Describe your incident:

I have this error message in server.log

ERROR [VersionProbe] Unable to retrieve version from indexer node: None of the TrustManagers trust this certificate chain. - None of the TrustManagers trust this certificate chain.

and this one in datanode.log

[OpensearchProcessImpl] Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

the datanode certificate has expired and now we are unable to access GUI with this error

the certificate for GUI in server.conf is still valid

2. Describe your environment:

• OS Information: Redhat 9.7

• Package Version: Graylog 6.3

server.conf:

datanode_tls_enabled = false
elasticsearch_ssl_verify = false
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = redacted
root_username = admin
root_password_sha2 = redacted
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = redacted:9000
http_publish_uri = https://redacted:9000
http_enable_tls = true
http_tls_cert_file = /etc/certificat/public.pem
http_tls_key_file = /etc/certificat/privkey.key
http_tls_key_password = redacted
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 20
outputbuffer_processors = 18
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
inputbuffer_processors = 5
message_journal_enabled = true
message_journal_dir = /data/graylog-server/journal
message_journal_max_age = 72h
message_journal_max_size = 80gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts

datanode.conf
node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = redacted
root_password_sha2 = redacted
mongodb_uri = mongodb://localhost/graylog
bind_address = 0.0.0.0
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /logdata/opensearch/data
opensearch_logs_location = /data/log/graylog-datanode/opensearch
opensearch_heap = 62g

3. What steps have you already taken to try and solve the problem?

i have tried to disable TLS by adding these to server.conf but it’s not working

datanode_tls_enabled = false
elasticsearch_ssl_verify = false

4. How can the community help?

Give any clue or idea to fix.

Many thanks

[Tdvorak (Tomas Dvorak)]

Hey @ajd

Are you using the default self-signed CA or your own CA?

Is your setup configured for automatic certificate renewal? If yes, restarting datanode service should be enough to trigger the renewal.

This mongodb query can confirm what you have configured:

graylog> db.cluster_config.find({‘type’:‘org.graylog2.plugin.certificates.RenewalPolicy’})
[
{
_id: ObjectId(“694a9e3a7ff1208fd9528efa”),
type: ‘org.graylog2.plugin.certificates.RenewalPolicy’,
payload: { mode: ‘AUTOMATIC’, certificate_lifetime: ‘P30D’ },
last_updated: ISODate(“2025-12-23T13:50:50.773Z”),
last_updated_by: ‘b7ee54e5-6ee4-404f-8e8c-b3d4ec40ac27’
}
]

Without GUI, you can still trigger the certificate renewal and restart of the datanode, by sending a POST request to http://server:port/api/certrenewal/your_datanode_node_id

Your node_id is persisted in the /etc/graylog/datanode/node-id file.

[ajd]

Hello, thank you for the answer

we use our own certificate for the GUI but we use the self-signed for the datanode

the configuration is set on MANUAL

_id: ObjectId(‘68d0fee8f91c805834337cc6’),
type: ‘org.graylog2.plugin.certificates.RenewalPolicy’,
payload: { mode: ‘MANUAL’, certificate_lifetime: ‘P30D’ },
last_updated: ISODate(‘2025-09-22T07:46:48.077Z’),
last_updated_by: ‘906fdddc-45d9-442a-966e-27575be4e9b6’

I have an error when i try this :

curl -X POST https://127.0.0.1:9200/api/certrenewal/2d822d0b-97b2-4e94-ab98-4e991dd81875 -k
{“error”:“no handler found for uri [/api/certrenewal/2d822d0b-97b2-4e94-ab98-4e991dd81875] and method [POST]”}

maybe it’s the wrong port but 9000 9300 and 8999 doesn’t give much response.

i have also tried deleting keystore.p12 in /var/lib/graylog-datanode/opensearch/config and there is a log asking me to log to the preflight interface but i have no longer access to it as in the graylog-server/server.log it doesn’t appear.

your api solution sounds good but it doesn’t work so far.

[ajd]

I finally found the solution

I have turned off both graylog-datanode and graylog-server then

I have changed the configuration in mongodb with the following command :

db.cluster_config.updateOne({ type: “org.graylog2.plugin.certificates.RenewalPolicy” },
{
$set: {
payload: {
mode: “AUTOMATIC”,
certificate_lifetime: “P365D”
}}})

then turn both services on again

then in mongodb i have found the rest API address at db.datanodes.find()

then curl https://server:8999/api/certrenewal/<node_id> (as you mentionned)

Thank you for your help