DataNode Migration stalled in step 3 Provision the Data Node's certificate

Graylog Central (peer support)
1

1. Describe your incident:

I try to do a Data Node Migration from openSearch 1.3.20 to the latest Data Node 7.1.5.

Everything runs smooth, but the process stalls in step 3 “Provision the Data Node’s certificate”

In the graylog server logs I find

Caused by: java.net.ConnectException: Failed to connect to barpbli26312239.prod.d001.loc/10.1.68.118:8999
at okhttp3.internal.connection.ConnectPlan.connectSocket(ConnectPlan.kt:278)
at okhttp3.internal.connection.ConnectPlan.connectTcp(ConnectPlan.kt:135)
at okhttp3.internal.connection.FastFallbackExchangeFinder$launchTcpConnect$1.runOnce(FastFallbackExchangeFinder.kt:141)
at okhttp3.internal.concurrent.TaskRunner$runnable$1.run(TaskRunner.kt:81)
... 3 more
Suppressed: java.net.ConnectException: Failed to connect to barpbli26312239.prod.d001.loc/10.1.68.118:8999
... 7 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:148)
at okhttp3.internal.connection.ConnectPlan.connectSocket(ConnectPlan.kt:276)
... 6 more
Suppressed: java.io.IOException: unexpected end of stream on ``https://barpbli26312239.prod.d001.loc:8999/``...

but if I telnet to the IP and the port (8999) I see no problem, telnet works.

2. Describe your environment:

  • OS Information: RHEL 8.10

  • Package Version: Graylog Version 7.1.5

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?

I modified the config.yml in the opensearch security configuration as recommended in the docs.

4. How can the community help?

Is there a way to get more detailed logs, especially on the data node?

Regards**,**

Dietmar

2

Hello,

I want to add some information/findings:

In the datanode.conf I defined this varibales:

opensearch_location = /usr/share/graylog-node/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /data/tools/opensearch/data
opensearch_logs_location = /data/logs/graylog-datanode

But the I get in the logfile /var/log/graylog-datanode/datanode.log this the error:

2026-07-14T08:48:05.530+02:00 ERROR [CmdLineTool] Startup error:
java.lang.IllegalArgumentException: Failed to list content of provided directory. Directory used for Opensearch detection: /usr/share/graylog-node/dist. Please configure opensearch_location to a directory that contains an opensearch distribution for your architecture x64. You can download Opensearch from ``https://opensearch.org/downloads.html`` . Please extract the downloaded distribution and point opensearch_location configuration option to that directory.

Looks like the opensearch installation can not be found. Opensearch was installed as a RPM packet: rpm -Uhv opensearch-1.3.20.rpm

and therefore should be in /usr/share/opensearch.

Whats wrong in my configuration. By the way, why does datanode write logs to /var/log/graylog-datanode if there is the line

opensearch_logs_location = /data/logs/graylog-datanode

I would expext logs in this directory.

Thanks in advance,

Dietmar Schurr

3

Hello @DietmarSchurr,

Graylog Data Node ships with it’s own distribution of Opensearch. It should be found here /usr/share/graylog-datanode/dist/opensearch-2.19.5-linux-x64/ as you noted above, can you confirm that the directory is empty.

4

Hello wine_Merchant,

thanks for your help.

The directory is not empty. The datanode-inlcuded Openserch is in the mentioned directory:

[OSEARCH03-T]root@server60:/usr/share/graylog-datanode/dist/opensearch-2.19.5-linux-x64>ls -al
total 972
drwxr-xr-x 9 root root 4096 Jul 9 08:52 .
drwxr-xr-x 3 root root 4096 Jul 8 14:38 ..
drwxr-xr-x 2 root root 4096 Jul 9 08:52 bin
drwxr-xr-x 5 root root 4096 Jul 9 08:52 config
drwxr-xr-x 9 root root 4096 Jul 9 08:52 jdk
drwxr-xr-x 3 root root 4096 Jul 9 08:52 lib
-rw-r--r-- 1 root root 11358 Jul 8 14:37 LICENSE.txt
drwxr-xr-x 2 root root 4096 Jul 8 14:38 logs
-rw-r--r-- 1 root root 9123 Jul 8 14:37 manifest.yml
drwxr-xr-x 22 root root 4096 Jul 8 14:38 modules
-rw-r--r-- 1 root root 924056 Jul 8 14:37 NOTICE.txt
-rwxr-xr-x 1 root root 2855 Jul 8 14:37 opensearch-tar-install.sh
drwxr-xr-x 17 root root 4096 Jul 8 14:38 plugins
-rw-r--r-- 1 root root 2709 Jul 8 14:37 README.md

So, my assumptions were wrong. The Graylog node is now running, the index has already been updated to version 9. (I can not go back to opensearch 1.3.20). But there is no connection to Graylog, and I think the problem is TLS and certificates. I defined in datanode.conf this variables:

transport_certificate = /etc/graylog/datanode/cacerts.jks
transport_certificate_password = changeme
http_certificate = /etc/graylog/datanode/cacerts.jks
http_certificate_password = changeme

Can I use absolute path in this variables?

What format does the certificate file can have? Java jks?

Thanks in advance for your help.

Regards,

Dietmar

5

@schurd the certificates should should be automatically created as a part of the migration to Data Node and shared to the Graylog node. Did you get as far as creating a CA and generating certificate for the Data Node?

The last step of the migration would be to comment out the elasticsearch_hosts option with your server.conf, were you prompted to do this?

6

Hello,

thanks for your help!

The step “provisioning certificate” was stalled (never ending), so I had to “Reset Migration”.

Is there a way to use our internal company-wide CA, and our internal company certificates?

This way we had TLS in all directions (Graylog ↔ opensearch, Graylog ↔ MongoDB, Graylog web interface) working before. We would prefer it this way with datanode as well.

The datanode documentation is not very verbose about the format of certificates and so on.

Regards,

Dietmar