Data Node Migration - Run directory compatibility check - fail

crickc · October 2, 2025, 8:12pm

Hello,

I’m running the Data Node Migration Wizard.

1. Describe your incident:

Data Node Migration [Run directory compatibility check] fail with a pop-up.

The Cluster Configuration node log for this node remain empty.

2. Describe your environment:

OS Information: Amazon Linux 2023
Package Version: 6.3.2
Service logs, configurations, and environment variables:

This is a small 1 Host custom Graylog installation with all services running on localhost (Graylog web interface is proxied). My Opensearch security plugins is disabled. Data-node is running with the same uid as Opensearch.

Datanode log file:

2025-10-02T18:12:57.367126+00:00 an-00003 graylog-datanode[2551430]: INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.24
2025-10-02T18:12:57.370709+00:00 an-00003 graylog-datanode[2551430]: INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/opt/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-10-02T18:14:06.142742+00:00 an-00003 graylog-datanode[2551430]: INFO  [OpensearchDataDirCompatibilityCheck] Found 424 indices and all of them are valid with current opensearch version 2.15.0
2025-10-02T18:14:06.733257+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeDirectories] Opensearch of the node 851633c1-320a-451d-af59-fb3ff76d6dc3 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/srv/opensearch/data', logsTargetDir='/var/log/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/srv/opensearch/config'}
2025-10-02T18:14:07.193396+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] Graylog Data Node 6.3.2+667aca0 starting up (command: datanode)
2025-10-02T18:14:07.194532+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] JRE: Eclipse Adoptium 17.0.16 on Linux 6.1.150-174.273.amzn2023.x86_64
2025-10-02T18:14:07.194532+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] Deployment: unknown
2025-10-02T18:14:07.195526+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] OS: Amazon Linux 2023 (amzn)
2025-10-02T18:14:07.195526+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] Arch: amd64
2025-10-02T18:14:07.262066+00:00 an-00003 graylog-datanode[2551430]: INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-10-02T18:14:07.263422+00:00 an-00003 graylog-datanode[2551430]: INFO  [PeriodicalsService] Delaying start of 0 periodicals until this node becomes leader ...
2025-10-02T18:14:07.264491+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-10-02T18:14:07.265551+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s].
2025-10-02T18:14:07.266165+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-10-02T18:14:07.267503+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-10-02T18:14:07.268120+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-10-02T18:14:07.294455+00:00 an-00003 graylog-datanode[2551430]: INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-10-02T18:14:07.360816+00:00 an-00003 graylog-datanode[2551430]: INFO  [SearchableSnapshotsConfigurationBean] Searchable snapshots are configured, adding opensearch configuration
2025-10-02T18:14:07.362586+00:00 an-00003 graylog-datanode[2551430]: INFO  [SearchableSnapshotsConfigurationBean] Search role enabled, validating usable space and adding search role to opensearch configuration
2025-10-02T18:14:07.398121+00:00 an-00003 graylog-datanode[2551430]: INFO  [TruststoreUtils] Detected existing JVM truststore: /opt/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-10-02T18:14:07.414741+00:00 an-00003 graylog-datanode[2551430]: INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/opt/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: INFO  [OpensearchProcessService]
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: ========================================================================================================
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: It seems you are starting Data node for the first time. The current configuration is not sufficient to
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: start the indexer process because a security configuration is missing. You have to either provide http
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.
2025-10-02T18:14:07.418631+00:00 an-00003 graylog-datanode[2551430]: ========================================================================================================
2025-10-02T18:14:07.419695+00:00 an-00003 graylog-datanode[2551430]: INFO  [JerseyService] Starting Data node REST API
2025-10-02T18:14:07.427808+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=36, OpensearchConfigurationService [RUNNING]=156}
2025-10-02T18:14:07.428513+00:00 an-00003 graylog-datanode[2551430]: INFO  [DatanodeBootstrap] Graylog DataNode datanode up and running.
2025-10-02T18:14:08.003216+00:00 an-00003 graylog-datanode[2551430]: INFO  [Version] HV000001: Hibernate Validator 8.0.2.Final
2025-10-02T18:14:08.383654+00:00 an-00003 graylog-datanode[2551430]: INFO  [NetworkListener] Started listener bound to [127.0.0.1:8999]
2025-10-02T18:14:08.385229+00:00 an-00003 graylog-datanode[2551430]: INFO  [HttpServer] [HttpServer] Started.
2025-10-02T18:14:08.385563+00:00 an-00003 graylog-datanode[2551430]: INFO  [JerseyService] Started REST API at <127.0.0.1:8999>

3. What steps have you already taken to try and solve the problem?

I could not find any error in the logs.

4. How can the community help?

Maybe someone here is knowledgeable enough about Graylog internals to help me debug this ?

Thanks !

Wine_Merchant · October 3, 2025, 9:04am

Hey @crickc,

Is there anything in the Graylog server.log?

crickc · October 3, 2025, 3:26pm

I really did a poor job at watching my logs for errors …

2025-10-03T14:56:36.846Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: java.io.InterruptedIOException: timeout
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:94) ~[graylog.jar:?]
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) ~[?:?]
	at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractTask.compute(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.runOnAllNodes(DatanodeRestApiProxy.java:89) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:121) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.request(DataNodeRestApiProxyResource.java:126) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.requestGet(DataNodeRestApiProxyResource.java:81) ~[graylog.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor438.invoke(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [graylog.jar:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.io.InterruptedIOException: timeout
	at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:209) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:148) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:91) ~[graylog.jar:?]
	... 44 more
Caused by: java.io.IOException: Canceled
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:72) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at org.graylog2.shared.bindings.providers.OkHttpClientProvider$UserAgentInterceptor.intercept(OkHttpClientProvider.java:78) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:148) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:91) ~[graylog.jar:?]
	... 44 more

I also found this line:

Suppressed: java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:8999

I changed the datanodebind_address to its default value and i’m getting the same error.

crickc · October 5, 2025, 4:13pm

I did a poor job at watching my logs for errors …

2025-10-03T14:56:49.930Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: java.io.InterruptedIOException: timeout
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:94) ~[graylog.jar:?]

There’s no firewall running here.

$ curl http://127.0.0.1:8999/
{"operating_system":{"os_name":"Linux","os_version":"6.1.150-174.273.amzn2023.x86_64","java_version":"17.0.16","user_name":"opensearch"},"opensearch":{"opensearch_version":"2.15.0","node":{"node_name":"localhost","state":"WAITING_FOR_CONFIGURATION","rest_base_url":"","process":{"pid":-1,"alive":false,"started":null}}},"datanode_directories":{"data_target_dir":"file:///srv/opensearch/data/","logs_target_dir":"file:///var/log/opensearch/","configuration_source_dir":"file:///etc/graylog/datanode/","configuration_target_dir":"file:///srv/opensearch/config/"},"dto":{"status":"UNCONFIGURED","error_msg":null,"cert_valid_until":null,"data_node_status":"UNCONFIGURED","cert_valid_until":null,"cluster_address":"localhost:9300","rest_api_address":"http://localhost:8999","action_queue":null,"datanode_version":"6.3.2+667aca0","opensearch_roles":[],"configuration_warnings":[],"version_compatible":true,"id":"851633c1-320a-451d-af59-fb3ff76d6dc3","node_id":"851633c1-320a-451d-af59-fb3ff76d6dc3","is_leader":false,"short_node_id":"851633c1","transport_address":"","hostname":"localhost","last_seen":"2025-10-05T16:06:33.000Z","object_id":"68dfe3e54f11ac0f048b7bf6","is_master":false},"data_node_version":"6.3.2+667aca0"}

Wine_Merchant · October 6, 2025, 8:27am

Could you post a sanitised version of the Datanode/Graylog node configuration files.

Within your browser tools, can you see that specific API call that is failing?

crickc · October 6, 2025, 3:17pm

Config:

# Graylog
is_leader = true
node_id_file = /srv/graylog/server/node-id
password_secret = REDACTED
root_password_sha2 = REDACTED
bin_dir = /opt/graylog/bin
data_dir = /srv/graylog/data
plugin_dir = plugin
http_external_uri = https://graylog.REDACTED/
stream_aware_field_types=false
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /srv/graylog/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://REDACTED@localhost:27017/graylog
mongodb_max_connections = 1000

# Datanode
node_id_file = /srv/opensearch/node-id
config_location = /etc/graylog/datanode
password_secret = REDACTED
root_password_sha2 = REDACTED
mongodb_uri = mongodb://REDACTED@localhost:27017/graylog
bind_address = 127.0.0.1
opensearch_location = /opt/graylog-datanode/dist/
opensearch_config_location = /srv/opensearch/config
opensearch_data_location = /srv/opensearch/data
opensearch_logs_location = /var/log/opensearch
opensearch_heap = 8g
path_repo = ["/srv/backup/opensearch/backup"]

In Firefox:

crickc · October 17, 2025, 9:47pm

Got http://127.0.0.1:8999/indices-directory/compatibility from tcpdump. A curl on this link take around a minute … I clearly need to increase some graylog timeout. Will test this next week.

system · October 31, 2025, 9:48pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Be a part of Graylog Development - Call for testers! Development data-node	12	213	August 30, 2024
Still can't seem to perform a Datanode migration Graylog Central (peer support) data-node	15	235	June 19, 2025
Graylog-datanode migration error Graylog Central (peer support) data-node	8	62	December 2, 2025
Failled to add new Datanode Graylog Central (peer support) data-node	25	384	June 5, 2025
Graylog fails to connect to Opensearch on port 9200 Graylog Central (peer support)	2	155	June 17, 2025

Data Node Migration - Run directory compatibility check - fail

Related topics