Data Node not showing up in Graylog 6.3 Preflight Configuration

douglas_ns · July 8, 2025, 1:38pm

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I’m setting up a Graylog 6.3 environment with a separate Data Node. However, the Data Node does not appear in the Graylog Server’s preflight configuration interface. Both servers can communicate with each other, MongoDB is correctly configured and accessible, and all services are running, but the Data Node is still not detected in the UI.

2. Describe your environment:
*Ubuntu 24

Package Version:
6.3
Service logs, configurations, and environment variables:
server.log
========================================================================================================

It seems you are starting Graylog for the first time. To set up a fresh install, a setup interface has
been started. You must log in to it to perform the initial configuration and continue.

Initial configuration is accessible at 192.168.200.253:9000, with username ‘admin’ and password ‘SLiwAgnYxj’.
Try clicking on http ://admin:SLiwAgnYxj@192.168.200.253:9000

========================================================================================================

(END)
datanode
2025-07-08T10:14:53.352-03:00 INFO [JerseyService] Starting Data node REST API
2025-07-08T10:14:53.352-03:00 INFO [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=68, OpensearchConfigurationService [RUNNING]=106}
2025-07-08T10:14:53.353-03:00 INFO [DatanodeBootstrap] Graylog DataNode datanode up and running.
2025-07-08T10:14:53.857-03:00 INFO [Version] HV000001: Hibernate Validator 8.0.2.Final
2025-07-08T10:14:54.119-03:00 INFO [NetworkListener] Started listener bound to [192.168.200.252:8999]
2025-07-08T10:14:54.121-03:00 INFO [HttpServer] [HttpServer] Started.
2025-07-08T10:14:54.121-03:00 INFO [JerseyService] Started REST API at <192.168.200.252:8999>

Curl from graylog-server node to datanode port 8999 returns
curl http ://elastick:8999
root@graylog:/var/log/graylog-server# curl http ://elastick:8999
{“operating_system”:{“os_name”:“Linux”,“os_version”:“5.15.0-143-generic”,“java_version”:“17.0.15”,“user_name”:“graylog-datanode”},“opensearch”:{“opensearch_version”:“2.15.0”,“node”:{“node_name”:“elastick”,“state”:“WAITING_FOR_CONFIGURATION”,“rest_base_url”:“”,“process”:{“pid”:-1,“alive”:false,“started”:null}}},“datanode_directories”:{“data_target_dir”:“file:///var/lib/graylog-datanode/opensearch/data/”,“logs_target_dir”:“file:///var/log/graylog-datanode/opensearch/”,“configuration_source_dir”:“file:///etc/graylog/datanode/”,“configuration_target_dir”:“file:///var/lib/graylog-datanode/opensearch/config/”},“dto”:{“status”:“UNCONFIGURED”,“error_msg”:null,“cert_valid_until”:null,“data_node_status”:“UNCONFIGURED”,“cert_valid_until”:null,“cluster_address”:“elastick:9300”,“rest_api_address”:“http://elastick:8999”,“action_queue”:null,“datanode_version”:“6.3.1+7bd8532”,“opensearch_roles”:,“configuration_warnings”:,“version_compatible”:true,“id”:“a1bd9815-d3ee-4b81-a156-c50312e4b001”,“is_leader”:false,“node_id”:“a1bd9815-d3ee-4b81-a156-c50312e4b001”,“short_node_id”:“a1bd9815”,“transport_address”:“”,“hostname”:“elastick”,“last_seen”:“2025-07-08T13:34:38.000Z”,“object_id”:“686d19cd4e2a9face703102c”,“is_master”:false},“data_node_version”:“6.3.1+7bd8532”}root@graylog:/var/log/graylog-server#

Tdvorak · July 9, 2025, 6:22am

Hi @douglas_ns,
If you don’t see any error messages, both services are running fine and the only symptom is that the datanode is not displayed in the preflight, then I’d suggest checking if both services are connected to the same mongodb server and the same database (=identical connection string).

The link between those two is always the mongodb, so if they happen to connect to different databases, they both think that they are alone in the cluster.

douglas_ns · July 9, 2025, 11:13am

Hi Tomas,
Thanks for your quick reply!

Here is my environment:

VM1: Graylog Server + MongoDB
IP: 192.168.200.253

VM2: Graylog Datanode
IP: 192.168.200.252

Both firewalls are disabled.

The MongoDB connection string is correctly set on both /etc/graylog/datanode/datanode.conf and /etc/graylog/server/server.conf as:

mongodb_uri = mongodb://192.168.200.253:27017/graylog

There are no errors in the logs on either machine.

When I run db.datanodes.find().pretty() in MongoDB, I get the following output:

[
{
_id: ObjectId(‘686d7b554e2a9face705adcb’),
node_id: ‘a1bd9815-d3ee-4b81-a156-c50312e4b001’,
datanode_status: ‘UNCONFIGURED’,
hostname: ‘elastick’,
is_leader: false,
last_seen: Timestamp({ t: 1752059203, i: 1 }),
transport_address: ‘’,
cluster_address: ‘elastick:9300’,
configuration_warnings: ,
datanode_version: ‘6.3.1+7bd8532’,
opensearch_roles: ,
rest_api_address: ‘hxxp://elastick:8999’
}
]

The hostname elastick resolves correctly to 192.168.200.252 from the Graylog server, and graylog resolves to 192.168.200.253 from the Datanode:

root@graylog:~# ping elastick
PING elastick (192.168.200.252) 56(84) bytes of data.
64 bytes from elastick (192.168.200.252): icmp_seq=1 ttl=64 time=9.29 ms
64 bytes from elastick (192.168.200.252): icmp_seq=2 ttl=64 time=10.2 ms

I even tried a fresh installation, removing both applications and the database, but the same issue happens again:
The datanode does not appear in the preflight configuration screen, and there are still no errors in the logs.

Any ideas on what might be missing?

Thanks in advance!

Tdvorak · July 10, 2025, 8:51am

Thank you for debugging. This all seems correct and working fine. Let’s focus on your browser. Can you check if you have any errors in the browser console? Anything blocked by the browser? Does the /api/data_nodes call return anything? Can you post a screenshot of the preflight page?

Thanks!

douglas_ns · July 10, 2025, 11:28am

Hi Tomas,

Thanks again!

I checked the browser console and there are no errors or blocked requests showing up.

Below are the screenshots of the preflight page as requested.

Let me know if there’s anything else I can check or provide.

Thanks!

curl -u admin:***** http://graylog:9000/api/data_nodes

Return:

I have a feeling it might be something simple that I’m overlooking.

Thanks!

Tdvorak · July 10, 2025, 12:11pm

This is indeed strange

I’d double check both graylog and datanode logs. If there is nothing wrong, I’d try to run the same mongo query from the graylog server. We know that datanode can reach it and store its information there, we want to verify that the server can too.

You can check if the cluster_config collection contains an entry with org.graylog2.bootstrap.preflight.PreflightEncryptedSecret type. This one is created by the server. So if both are connected to the same DB, you’ll see an entry in the datanode collection and an the encrypted secret in the cluster_config collection.

You can also observe if the last_seen field of the datanode entry gets updated - if the datanode is running and pinging/updating the db.

Just to be sure - what mongodb version are you using?

Tdvorak · July 10, 2025, 12:21pm

One more question - which version of the graylog server are you running, please?

douglas_ns · July 10, 2025, 12:57pm

mongodb version is 7.0.21

server.log

datanode.log

Hi Tomas,

Thanks again for your help!

I followed your suggestions and connected to MongoDB using mongosh from the Graylog Server. Here are the results:

The cluster_config collection contains the entry of type org.graylog2.bootstrap.preflight.PreflightEncryptedSecret, created by the server:

{
type: “org.graylog2.bootstrap.preflight.PreflightEncryptedSecret”,
last_updated_by: “a1bd9815-d3ee-4b81-a156-c50312e4b001”
}

The datanodes collection also contains the expected entry.
The last_seen field is being updated, indicating the datanode is alive and communicating with MongoDB.
However, it seems that the last_seen has not been updated for some time.

{
node_id: “a1bd9815-d3ee-4b81-a156-c50312e4b001”,
datanode_status: “UNCONFIGURED”,
last_seen: Timestamp({ t: 1752151703, i: 1 }), // corresponds to 2025-07-09 08:28:23 (GMT-3)
hostname: “elastick”,
rest_api_address: “http://elastick:8999”
}

So it appears that both the server and the datanode are connected to the same MongoDB database and are writing data correctly.

Please let me know if there is anything else I should check. Thanks again for your help!

Tdvorak · July 10, 2025, 1:38pm

Thanks, I think we are getting close. The last_seen timestamp should be fairly recent - the datanode should update its data every second with a periodical task. If it’s really outdated, then maybe the task/datanode stopped running? Your logs in the screenshots show old timestamps too, is it just an old screenshot or are there really no new logs?

If the last_seen is that outdated, it will be considered invalid and ignored in the preflight, where we consider only recent entries.

I’d try to restart the datanode service and check if the last_seen is getting updated.

douglas_ns · July 10, 2025, 1:46pm

Hi Tomas,

I restarted the Datanode as suggested, and checked the last_seen field again.

Tdvorak · July 10, 2025, 1:52pm

Ok, that looks good, the periodical is running, the timestamp is getting updated, logs are having recent timestamp as well. Does it help? Do you see the datanode in the preflight now?

douglas_ns · July 10, 2025, 2:03pm

Hi Tomas,

Thanks for the follow-up!

The last_seen timestamp is indeed updating now, and the logs are showing recent entries — so the periodic task seems to be working properly.

However, the datanode still does not appear in the preflight UI.

At this point, I’m considering reinstalling both VMs from scratch with a different operating system, just to rule out any OS-level issue or misconfiguration.

Before I go down that path — is there anything else we can try or verify?

Thanks again for your continued support!

Tdvorak · July 10, 2025, 2:27pm

One more question - based on the look of the logo in the preflight setup, it seems that you are running an older version of the server. Can you please verify that your graylog server is also 6.3 version?

douglas_ns · July 10, 2025, 2:39pm

Hi Tomas,

Strangely, the Graylog Server version installed is 5.2.12

I followed this guide for the installation:

I apologize for not noticing this earlier. Anyway, you have identified the problem, and I’m grateful for your attention and help.

Tdvorak · July 10, 2025, 2:49pm

Good to hear that we found the problem!

One of my QA colleagues, @vadym.vasylenko, noticed the old logo and asked about the sever version. Thanks Vadym!

With the latest 6.3 version, does your datanode appear in the preflight, as it should?

douglas_ns · July 10, 2025, 3:42pm

Yes, after installing version 6.3, everything is working perfectly — the datanode now shows up in the preflight just as expected.

Huge thanks to you, Tomas, and @vadym.vasylenko for your time, patience, and for spotting the version mismatch.

Apologies for the confusion — I’m honestly a bit embarrassed
But I truly learned a lot from this experience. Thank you again!

Tdvorak · July 11, 2025, 6:26am

Thank you for the confirmation @douglas_ns! No need to be embarrassed, it was a tricky situation. We are also learning from your experience, so thank you too!

Best regards,
Tomas

Topic		Replies	Views
Preflight: Unconfigured datanode appears once in list, then disappears Graylog Central (peer support)	8	69	June 19, 2025
No data nodes have been found after installation Graylog Central (peer support)	8	1536	December 10, 2024
Pre-flight Interface cannot see Datanode Graylog Central (peer support)	4	133	May 5, 2025
Data nodes have not been found Graylog Central (peer support) data-node	4	83	June 5, 2025
PrePreflight error on new installation (Graylog 6.1) Unable to retrieve version from indexer node Graylog Central (peer support)	1	189	March 19, 2025

Data Node not showing up in Graylog 6.3 Preflight Configuration

Related topics