Graylog Initial Setup fails to provision CA to datanode

[stevenwhitej (steven white)]

1. Describe your incident:
I am using Docker to deploy graylog, 1 datanode, elasticsearch, and mongodb. After running “docker compose up -d”, I am able to access the Graylog Initial Setup screen, where I:

1. See the datanode is visible in the UI
2. Create a CA
3. Create the CA policy
4. Provision certificate and continue, to push the CA to the datanode
5. The Initial Setup UI reports this error on the datanode: “TOO MANY FOLLOW-UP REQUESTS: 21”

2. Describe your environment:
I am running Docker on Windows with WSL2. This is my docker-compose.yml:
version: ‘2’
services:
mongodb: # Mongodb service
image: mongo #
volumes:
- mongo_data:/data/db # Persisted mongodb data
elasticsearch: # ElasticSearch service
image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0 # Version of elasticsearch image
volumes:
- es_data:/usr/share/elasticsearch/data # Persisted elasticsearch data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
# Unlimited amount of memory set for elasticsearch container
memlock:
soft: -1
hard: -1
mem_limit: 1g
graylog:
image: graylog/graylog:5.2.3 # Version of Graylog docker image
volumes:
- graylog_data:/usr/share/graylog/data # Persisted Graylog data
environment:
- GRAYLOG_PASSWORD_SECRET=“<snipped”
- GRAYLOG_ROOT_PASSWORD_SHA2=“<snipped”
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
- GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
entrypoint: /usr/bin/tini – wait-for-it elasticsearch:9200 – /docker-entrypoint.sh
links:
- mongodb:mongo
- elasticsearch
restart: always
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
datanode1:
image: graylog/graylog-datanode:5.2.3
hostname: “datanode1”
depends_on:
- mongodb
environment:
GRAYLOG_DATANODE_NODE_ID_FILE: “/var/lib/graylog-datanode/node-id”
GRAYLOG_DATANODE_PASSWORD_SECRET: “<snipped”
GRAYLOG_DATANODE_ROOT_PASSWORD_SHA2: “<snipped”
GRAYLOG_DATANODE_MONGODB_URI: “mongodb://mongodb:27017/graylog”
ulimits:
memlock:
hard: -1
soft: -1
nofile:
soft: 65536
hard: 65536
ports:
- “8999:8999/tcp” # DataNode API
volumes:
- “graylog-datanode-01:/var/lib/graylog-datanode”
restart: “on-failure”
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_data:
driver: local
graylog-datanode-01:
driver: local

• Service logs, configurations, and environment variables:
  I checked the logs from the graylog container:
  2024-02-02 16:03:35,185 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
  2024-02-02 16:03:35,185 INFO : org.graylog2.security.CustomCAX509TrustManager - CA changed, refreshing trust manager
  2024-02-02 16:04:36,737 INFO : org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical - Starting connectivity check with node 8fdf6c84-dce1-43cf-813c-1d487ed4e59b, silencing error messages for 60 seconds.
  2024-02-02 16:05:38,236 WARN : org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical - Error trying to connect to data node 8fdf6c84-dce1-43cf-813c-1d487ed4e59b: Too many follow-up requests: 21, retrying (attempt #21)

And the Graylog datanode container logs show:
2024-02-02T16:03:35.879Z INFO [CustomCAX509TrustManager] CA changed, refreshing trust manager
2024-02-02T16:03:36.168Z ERROR [CustomCAX509TrustManager] Could not add Graylog CA to TrustManagers: Failed to load keystore from Mongo collection for node GRAYLOG CA
org.graylog.security.certutil.ca.exceptions.KeyStoreStorageException: Failed to load keystore from Mongo collection for node GRAYLOG CA
at org.graylog.security.certutil.keystore.storage.KeystoreMongoStorage.readKeyStore

3. What steps have you already taken to try and solve the problem?
I see connections from the datanode to the mongo instance, so I don’t think its network related. But, I’m not sure how to proceed in isolating the datanode to mongo issue.

4. How can the community help?
I need help to understand how to successfully update the trust manager on the datanode, or suggestions on how to workaround this issue to ensure communications are still secure.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]