DataNode node name changes everytime it restarts

1. Describe your incident:

Using Podman/Quadlet to implement Graylog 6.1. Mapped persistent data for the DataNode using this command:

Volume=/graylog-data/datanode:/var/lib/graylog-datanode:U,Z

On the first start, everything successful and I was able to successfully perform the PreFlight configuration and Graylog could successfully connect to the DataNode.

However, each restart the DataNode rebuilds the opensearch.yml file and the node.name changes, doesn’t match the hostname of the certificate that Graylog is using to communicate with OpenSearch, and then Graylog ends up dying.

I’ve been researching how to persist that information, but have yet to find anything. Would appreciate being pointed in the right direction.

2. Describe your environment:

  • OS Information:
    Red Hat Enterprise Linux release 9.5 (Plow)
  • Package Version:
    Graylog containers, v6.1
  • Service logs, configurations, and environment variables:

Successful start in DataNode aside from these errors when Graylog attempts to connect:

2025-01-30T16:19:54.454703556-08:00 2025-01-31T00:19:54.454Z WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 443 unavailable. Cause: Host name ‘9862632e5d56’ does not match the certificate subject provided by the peer (CN=299cc13753ca)

299cc13753ca is the original name of the node.name, while the first Host name ‘XXXXXXXX’ value continues to change.

Logs from the perspective of the Graylog container:

2025-01-30T16:22:50.700405569-08:00 00:22:50.700 [main] ERROR org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from indexer node: Hostname 9862632e5d56 not verified:
2025-01-30T16:22:50.700405569-08:00 certificate: sha256/HBjcDialxDmnJCgMGTVbd81a5mSfLhQBnDfiDvYjHss=
2025-01-30T16:22:50.700405569-08:00 DN: CN=299cc13753ca
2025-01-30T16:22:50.700405569-08:00 subjectAltNames: [127.0.0.1, 0:0:0:0:0:0:0:1, 10.89.0.31, localhost, 299cc13753ca]. - Hostname 9862632e5d56 not verified:
2025-01-30T16:22:50.700405569-08:00 certificate: sha256/HBjcDialxDmnJCgMGTVbd81a5mSfLhQBnDfiDvYjHss=
2025-01-30T16:22:50.700405569-08:00 DN: CN=299cc13753ca
2025-01-30T16:22:50.700405569-08:00 subjectAltNames: [127.0.0.1, 0:0:0:0:0:0:0:1, 10.89.0.31, localhost, 299cc13753ca].

3. What steps have you already taken to try and solve the problem?

I’ve validated that I’ve set the volumes for persisting data as per the documentation.

4. How can the community help?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hello @neteng45,

Could you post your full configuration?

Are you using ‘GRAYLOG_DATANODE_NODE_NAME’ ?

If no explicit node name is defined, data node will fallback to hostname. And your hostname seems to be changing with every restart of the service. So I’d recommend fixing the hostname by the configuration @Wine_Merchant mentions. You’ll need to make sure that this name is accessible from the graylog server.

1 Like

@Wine_Merchant @Tdvorak
Thank you both for taking the time to reply.

I was expecting that my container hostname take on the name of the container and not the container ID but it appears that it took the container ID and I somehow managed to overlook that fact when troubleshooting, my apologies.

You both were spot on, once I set the node name via GRAYLOG_DATANODE_NODE_NAME and then also hardcoded the hostname in my Podman Quadlet container file, the errors were resolved without issue. I’ll ensure I do that going forward as I finish building out the cluster.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.