Graylog Cluster installation : Read timed out. - Read timed out

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I’m installing a graylog cluster, with 3 datanodes, when I launch the graylog server (the datanodes and server are on differents VM) I end up with this error:
2025-02-13T17:15:06.057+01:00 INFO [VersionProbe] Indexer is not available. Retry #46
2025-02-13T17:15:21.081+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Read timed out. - Read timed out.
2025-02-13T17:15:31.101+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Read timed out. - Read timed out.
2025-02-13T17:15:41.121+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Read timed out. - Read timed out.

The VMs can ping each other, can curls each other on the API port,
when I curl on one the datanode I have this result:
{“operating_system”:{“os_name”:“Linux”,“os_version”:“5.14.0-503.23.2.el9_5.x86_64”,“java_version”:“17.0.14”,“user_name”:“graylog-datanode”},“opensearch”:{“opensearch_version”:“2.15.0”,“node”:{“node_name”:“prlaps01graylog-datanode”,“state”:“WAITING_FOR_CONFIGURATION”,“rest_base_url”:“”,“process”:{“pid”:-1,“alive”:false,“started”:null}}},“datanode_directories”:{“data_target_dir”:“file:///nfs/graylog/prl01graylog-datanode/”,“logs_target_dir”:“file:///var/log/graylog-datanode/opensearch/”,“configuration_source_dir”:“file:///etc/graylog/datanode/”,“configuration_target_dir”:“file:///var/lib/graylog-datanode/opensearch/config/”,“opensearch_process_configuration_dir”:“file:///var/lib/graylog-datanode/opensearch/config/opensearch/”,“data_target_dir_space”:“2 TB”,“logs_target_dir_space”:“3 GB”,“configuration_target_dir_space”:“32 GB”},“data_node_version”:“6.1.6+a644883”}[

Mongo’s log doesn’t pinpoint anything

2. Describe your environment:

  • OS Information:
    RHEL 9

  • Package Version:
    graylog-6.1.6

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
already tried to restart servers with no success, since the errors only said read timed out I’m not sure of what to try
the curl reveal that it’s waiting for configuration but since I’m supposed to configure it with preflight, I’m not sure if it is common behavior or not

I’ve already looked for firewall issue; it is not.

4. How can the community help?
Any new idea would be gladly received

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hi @Leo_S,
You are configuring a completely new cluster, right? Then your graylog server should give you the preflight configuration interface where you configure a CA and provision certificates for your data nodes. Only then they can start. Currently your data nodes are waiting for this provisioning (WAITING_FOR_CONFIGURATION state you see in their APIs).

If you don’t see the preflight interface in your graylog server and get Unable to retrieve version from indexer node: Read timed out. - Read timed out, then I assume you have configured something elasticsearch_hosts in your graylog server.conf. Could it be? If yes, then please remove this configuration option, restart your graylog server and you should get the preflight interface.

Best regards,
Tomas

1 Like

Thank you very much, it was indeed the elasticsearchhost option that was creating this issue.

If I understand well, since I am using Graylog Datanode, I do not need to use elasticsearch_hosts, and the server finds the datanode thanks to MongoDB ?

1 Like

Exactly, the graylog server is autodetecting data nodes. They are registered in the mongodb and their address is published there.

The elasticsearch_hosts option is a legacy one, to support old elasticsearch and opensearch nodes. When provided, it blocks the autodetection, as you have seen.

Nice to hear that everything is working for you now!

Best regards,
Tomas

1 Like