1. Describe your incident:
A few months ago I made a fresh deployment of Graylog Open 6.3 with Docker Compose while following the relevant documentation. I configured the internal Graylog CA and provisioned the Data Node certificate using the preflight interface, then used my own CA to secure the web interface, and even managed to restore old configs and several hundred GBs of data from an earlier deployment. Everything went smoothly.
So a few days ago I upgraded to version 7.0, then to 7.1, by doing docker compose down → updating the image tags in .env → docker compose up. All seemed well after the 7.0 update. But when version 7.1 went up, I saw this in the Graylog container logs (see PDF for full context):
graylog_server_logs_after_v7.1.pdf (58.1 KB)
2026-07-06 17:14:14,176 ERROR: org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from indexer node: Hostname 172.18.0.4 not verified:
certificate: sha256/qiZf/I2AUsNZo0ORP/iFfOA6WfC+WfCplyHr6XdyMtc=
DN: CN=datanode
subjectAltNames: [127.0.0.1, 0:0:0:0:0:0:0:1, 172.18.0.2, localhost, datanode]. - Hostname 172.18.0.4 not verified:
certificate: sha256/qiZf/I2AUsNZo0ORP/iFfOA6WfC+WfCplyHr6XdyMtc=
DN: CN=datanode
subjectAltNames: [127.0.0.1, 0:0:0:0:0:0:0:1, 172.18.0.2, localhost, datanode].
2026-07-06 17:14:14,178 INFO : org.graylog2.storage.versionprobe.VersionProbe - Indexer is not available. Retry #1/1
2026-07-06 17:14:14,178 ERROR: org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from indexer node:
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 1 attempts.
at com.github.rholder.retry.Retryer.call(Retryer.java:174)
at org.graylog2.storage.versionprobe.VersionProbeImpl.probe(VersionProbeImpl.java:116)
at org.graylog2.periodical.SearchVersionCheckPeriodical.doRun(SearchVersionCheckPeriodical.java:113)
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99)
at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedPeriodicRunnable.run(InstrumentedScheduledExecutorService.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
And then the three VersionProbe messages started repeating every 30s. This is flooding the logs, but I can’t find any issues with the services. Data Node logs show no errors. Inputs, searches, index maintenance etc are all working as expected. Also: curl -k https[://]datanode[:]9200 works from inside the Graylog container, so I guess it can resolve the datanode hostname without issue.
2. Describe your environment:
-
OS Information: Docker Compose on AlmaLinux 9
-
Package Version: 7.1 for Graylog and Data Node, 7.0.29 for MongoDB
-
Service logs, configurations, and environment variables:
.env
GRAYLOG_IMAGE="graylog/graylog:7.1"
DATANODE_IMAGE="graylog/graylog-datanode:7.1"
MONGODB_IMAGE="mongo:7.0.29"
docker-compose.yml
services:
mongodb:
image: "${MONGODB_IMAGE:-mongo:7.0.29}"
restart: always
networks:
- graylog
volumes:
- "mongodb_data:/data/db"
- "mongodb_config:/data/configdb"
environment:
- TZ=[my-time-zone]
datanode:
image: "${DATANODE_IMAGE:-graylog/graylog-datanode:6.3}"
hostname: "datanode"
environment:
GRAYLOG_DATANODE_NODE_ID_FILE: "/var/lib/graylog-datanode/node-id"
GRAYLOG_DATANODE_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET}"
GRAYLOG_DATANODE_MONGODB_URI: "mongodb://mongodb:27017/graylog"
GRAYLOG_DATANODE_OPENSEARCH_HEAP: "16g"
GRAYLOG_DATANODE_OPENSEARCH_CONFIGURATION_OVERRIDES_FILE: "/var/lib/graylog-datanode/overrideconfig.conf"
TZ: "[my-time-zone]"
ulimits:
memlock:
hard: -1
soft: -1
nofile:
soft: 65536
hard: 65536
ports:
- "127.0.0.1:8999:8999/tcp"
- "127.0.0.1:9200:9200/tcp"
networks:
- graylog
volumes:
- "datanode_data:/var/lib/graylog-datanode"
restart: always
graylog:
hostname: "server"
image: "${GRAYLOG_IMAGE:-graylog/graylog:6.3}"
depends_on:
mongodb:
condition: "service_started"
datanode:
condition: "service_started"
entrypoint: "/usr/bin/tini -- /docker-entrypoint.sh"
environment:
GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
GRAYLOG_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET}"
GRAYLOG_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2}"
GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
GRAYLOG_HTTP_EXTERNAL_URI: "https://my.graylog.url:9000/"
GRAYLOG_HTTP_PUBLISH_URI: "https://my.graylog.url:9000/"
GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
GRAYLOG_ROOT_TIMEZONE: "[my-time-zone]"
TZ: "[my-time-zone]"
GRAYLOG_HTTP_ENABLE_TLS: true
GRAYLOG_HTTP_TLS_CERT_FILE: "/usr/share/graylog/tls/graylog.pem"
GRAYLOG_HTTP_TLS_KEY_FILE: "/usr/share/graylog/tls/graylog.key"
GRAYLOG_SERVER_JAVA_OPTS: "-Xms8g -Xmx8g -Djavax.net.ssl.trustStore=/usr/share/graylog/tls/graylog.jks"
ports:
- "9000:9000/tcp" # Server API
- "...[my-input-ports]..."
networks:
- graylog
volumes:
- "graylog_data:/usr/share/graylog/data"
- "graylog_tls:/usr/share/graylog/tls"
restart: always
networks:
graylog:
driver: "bridge"
volumes:
mongodb_data:
external: true
mongodb_config:
external: true
datanode_data:
external: true
graylog_data:
external: true
graylog_tls:
external: true
3. What steps have you already taken to try and solve the problem?
I tried renewing the Data Node certificate using the web interface - the operation completed successfully but nothing changed. I did a few rounds of docker compose down / up and even rebooted the host. The errors stopped showing up this one time - I guess the new IP assigned to the Data Node container at that time matched the value the server was looking for. But I have had to reboot the host more times since and the errors always come back.
I searched for similar cases, found a few reports of VersionProbe errors (like 1, 2, 3 and 4), but nothing quite like this.
4. How can the community help?
My speculation: the preflight routine stored in MongoDB (or somewhere) the IP value that the datanode container had at that time, and now VersionProbe won’t use the container hostname when connecting to Data Node, just that one frozen IP, even though the hostname is included in the certificate SAN field.
Has anyone else seen this in their logs? Is this a bug? Do I need to freeze my container IPs (I would rather not to, if possible)? Can I make Graylog “forget” that one IP value and use the Data Node container hostname (configured in the Compose file) every time?