1. Describe your incident:
- graylog-datanode container:
- certificate is expired
- automatic certificate renewal does not work
- log:
...
2026-06-08T07:18:39.710Z INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 10. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb:27017, type=UNKNOWN, state=CONNECTING}].
2026-06-08T07:18:39.712Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2958994, minRoundTripTimeNanos=0}
2026-06-08T07:18:39.751Z INFO [MongoDBPreflightCheck] Connected to MongoDB version 7.0.34
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
2026-06-08T07:18:39.873Z INFO [MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2026-06-08T07:18:40.065Z WARN [VectorizationProvider] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
2026-06-08T07:18:40.552Z INFO [OpensearchDataDirCompatibilityCheck] Found 47 indices and all of them are valid with current opensearch version 2.19.3
2026-06-08T07:18:40.749Z INFO [DatanodeKeystoreInitService] Datanode certificate expired on 2026-05-14T08:48:56.000+0000. Requesting and awaiting new certificate
2026-06-08T07:18:40.930Z INFO [CsrRequesterImpl] Triggered certificate signing request for this datanode
2026-06-08T07:18:40.949Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #1
2026-06-08T07:18:45.979Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #2
2026-06-08T07:18:51.007Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #3
2026-06-08T07:18:56.036Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #4
2026-06-08T07:19:01.065Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #5
2026-06-08T07:19:06.096Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #6
2026-06-08T07:19:11.125Z INFO [DatanodeKeystoreInitService] Waiting for datanode certificate renewal, retry #7
...
<this goes on forever>
...
- graylog-mongodb container:
- does start
- log:
t={"$date":"2026-06-08T10:17:45.475+00:00"} s=I c=WTCHKPT id=22430 ctx=Checkpointer msg=WiredTiger message attr={"message":{"ts_sec":1780913865,"ts_usec":475702,"thread":"1:0x7f01a8a9c640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 80, snapshot max: 80 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 12096805"}}
- graylog-graylog (server) container
- fails to start because it can’t connect to graylog-datanode
- log:
2026-06-08 10:17:17,464 ERROR: org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from indexer node: Failed to connect to datanode/172.18.0.2:9200. - Connection refused.
2026-06-08 10:17:17,465 INFO : org.graylog2.storage.versionprobe.VersionProbe - Indexer is not available. Retry #582
- datanode api is not accessible from graylog-graylog, so:
curl https://datanode:8999/api/certrenewal/<node_id>is not possible
2. Describe your environment:
-
OS Information: debian 13 with Docker version 29.4.2
-
Package Version: graylog-open 7.0 / 7.1
-
single instance
-
docker compose
- Using the docker-compose open template from: docker-compose/open-core/docker-compose.yml at main · Graylog2/docker-compose · GitHub
-
Service logs, configurations, and environment variables:
docker-compose.yml
services:
mongodb:
hostname: "mongodb"
image: "mongo:7.0"
restart: "on-failure"
networks:
- graylog
volumes:
- "mongodb_data:/data/db"
- "mongodb_config:/data/configdb"
# ⚠️ Make sure this is set on the host before starting:
# echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
# sudo sysctl -p
datanode:
image: "${DATANODE_IMAGE:-graylog/graylog-datanode:7.1}"
hostname: "datanode"
environment:
GRAYLOG_DATANODE_NODE_ID_FILE: "/var/lib/graylog-datanode/node-id"
# GRAYLOG_DATANODE_PASSWORD_SECRET and GRAYLOG_PASSWORD_SECRET MUST be the same value
GRAYLOG_DATANODE_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
GRAYLOG_DATANODE_MONGODB_URI: "mongodb://mongodb:27017/graylog"
GRAYLOG_DATANODE_OPENSEARCH_HEAP: "16g"
#GRAYLOG_DATANODE_INSECURE_STARTUP: true # tls cert fix, unsupported ?
ulimits:
memlock:
hard: -1
soft: -1
nofile:
soft: 65536
hard: 65536
#ports:
# - "8999:8999/tcp" # DataNode API
# - "9200:9200/tcp"
# - "9300:9300/tcp"
networks:
- graylog
volumes:
- "graylog-datanode:/var/lib/graylog-datanode"
restart: "on-failure"
graylog:
hostname: "server"
image: "${GRAYLOG_IMAGE:-graylog/graylog:7.1}"
depends_on:
mongodb:
condition: "service_started"
datanode:
condition: "service_started"
entrypoint: "/usr/bin/tini -- /docker-entrypoint.sh"
environment:
GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
GRAYLOG_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
GRAYLOG_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2:?Please configure GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file}"
GRAYLOG_TIMEZONE: "Europe/Berlin"
GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
#GRAYLOG_HTTP_PUBLISH_URI: "https://graylog.example.com/"
#GRAYLOG_HTTP_EXTERNAL_URI: "http://127.0.0.1:9000/"
GRAYLOG_HTTP_EXTERNAL_URI: "https://graylog.example.com/"
#GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch:9200"
GRAYLOG_ELASTICSEARCH_HOSTS: "https://datanode:9200"
GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
GRAYLOG_TRANSPORT_EMAIL_ENABLED: "true"
GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: "mail.example.com"
GRAYLOG_TRANSPORT_EMAIL_FROM_EMAIL: "graylog@example.com"
GRAYLOG_TRANSPORT_EMAIL_PORT: "25"
GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"
GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"
GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"
#GRAYLOG_TRUSTED_PROXIES: "127.0.0.1/32, 172.18.0.0/16"
ports:
- "514:514/udp" # Syslog
- "514:514/tcp" # Syslog
#- "5044:5044/tcp" # Beats
- "5141:5141/udp" # Input
- "5141:5141/tcp" # Input
- "5142:5142/udp" # Input
- "5142:5142/tcp" # Input
- "5143:5143/udp" # Input
- "5143:5143/tcp" # Input
- "5144:5144/udp" # Input
- "5144:5144/tcp" # Input
- "5145:5145/udp" # Input
- "5146:5146/udp" # Input
- "5147:5147/udp" # Input
#- "5555:5555/tcp" # Input
#- "5555:5555/udp" # Input
#- "5001:5001/udp" # Firewall
#- "9000:9000/tcp" # Server API
#- "12201:12201/tcp" # GELF TCP
#- "12201:12201/udp" # GELF UDP
#- "10000:10000/tcp" # Custom TCP port
#- "10000:10000/udp" # Custom UDP port
#- "13301:13301/tcp" # Forwarder data
#- "13302:13302/tcp" # Forwarder config
networks:
- graylog
- caddy
volumes:
- "graylog_data:/usr/share/graylog/data/data"
restart: "on-failure"
networks:
graylog:
name: "graylog"
driver: "bridge"
caddy:
external: true
volumes:
mongodb_data:
mongodb_config:
graylog-datanode:
graylog_data:
3. What steps have you already taken to try and solve the problem?
- Verified org.graylog2.plugin.certificates.RenewalPolicy is set to automatic: Unable to access GUI after datanode certificate expire
- Tried every solution I could find in the community
- Search the docs, community, google, asked ai, github issues → no hint or answer whats wrong or how to fix the issue
4. How can the community help?
Troubleshoot the issue