Data Node Migration Failure

Reset8400 · November 13, 2025, 9:26pm

1. Describe your incident: When trying to migrate from opensearch to data node, I get a constant 401 Unauthorized after clicking next on Step 5 (Stop message processing)

2. Describe your environment:

OS Information:Ubuntu 24.04
Package Version: graylog/datanode 7.0.0-10, opensearch 2.19.3
Service logs, configurations, and environment variables:

server.log

2025-11-13T12:18:31.838-05:00 INFO [MigrationActionsImpl] Attempting to pause processing on all nodes…
2025-11-13T12:18:31.839-05:00 INFO [ClusterProcessingControl] Attempting to call ‘pause processing’ on node [348c4e60-fb43-4691-a683-6bc96da5e328].
2025-11-13T12:18:31.841-05:00 INFO [SystemProcessingResource] Paused message processing - triggered by REST call.
2025-11-13T12:18:31.841-05:00 INFO [ClusterProcessingControl] Successfully called ‘pause processing’ on node [348c4e60-fb43-4691-a683-6bc96da5e328].
2025-11-13T12:18:31.841-05:00 INFO [MigrationActionsImpl] Done pausing processing on all nodes.
2025-11-13T12:18:31.841-05:00 INFO [MigrationActionsImpl] Waiting for output buffer to drain on all nodes…
2025-11-13T12:18:31.842-05:00 INFO [ClusterProcessingControl] Attempting to call ‘fetching output rate metric value’ on node [348c4e60-fb43-4691-a683-6bc96da5e328].
2025-11-13T12:18:31.844-05:00 INFO [ClusterProcessingControl] Successfully called ‘fetching output rate metric value’ on node [348c4e60-fb43-4691-a683-6bc96da5e328].
2025-11-13T12:18:31.844-05:00 INFO [ClusterProcessingControl] Output buffer is now empty on all nodes.
2025-11-13T12:18:31.844-05:00 INFO [MigrationActionsImpl] Done waiting for output buffer to drain on all nodes.
2025-11-13T12:18:31.889-05:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.IllegalStateException: Failed to trigger datanode request. Code: 401, message: Unauthorized
at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$remoteInterface$9(DatanodeRestApiProxy.java:172)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source)
at java.base/java.util.stream.AbstractTask.compute(Unknown Source)
at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.remoteInterface(DatanodeRestApiProxy.java:157)
at org.graylog.plugins.views.storage.migration.state.actions.MigrationActionsImpl.isOldClusterStopped(MigrationActionsImpl.java:124)
at com.github.oxo42.stateless4j.triggers.TriggerBehaviour.isGuardConditionMet(TriggerBehaviour.java:31)
at com.github.oxo42.stateless4j.StateRepresentation.getPermittedTriggers(StateRepresentation.java:179)
at com.github.oxo42.stateless4j.StateMachine.getPermittedTriggers(StateMachine.java:128)
at org.graylog.plugins.views.storage.migration.state.machine.MigrationStateMachineImpl.nextSteps(MigrationStateMachineImpl.java:79)
at org.graylog.plugins.views.storage.migration.state.machine.MigrationStateMachineImpl.trigger(MigrationStateMachineImpl.java:59)
at org.graylog.plugins.views.storage.migration.state.rest.MigrationStateResource.trigger(MigrationStateResource.java:67)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

server.conf

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = <secret>
root_password_sha2 = <sha2>
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts

datanode.conf

node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = <same secret>
root_password_sha2 = <same sha2>
mongodb_uri = mongodb://localhost/graylog
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /data/opensearch
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 4g

opensearch.yml

cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /data/opensearch
path.logs: /var/log/opensearch
network.host: 127.0.0.1
discovery.type: single-node
action.auto_create_index: false
plugins.security.disabled: true

3. What steps have you already taken to try and solve the problem?

Re-install datanode, verify passwords match, verify configs, restart services, rebooting

4. How can the community help?

Please help me fix my error so I can migrate off opensearch

Tdvorak · November 18, 2025, 7:02am

Hi @Reset8400 ,

Thank you for reporting this. There is a known bug, which has been already fixed and will be part of the first bugfix release 7.0.1, scheduled for this week.

Meanwhile, you could try a workaround. In your graylog.conf (the server configuration), you can set indexer_use_jwt_authentication=true, which will enforce usage of jwt auth headers in server → datanode communication and should mitigate the problem you observe.

Please let me know if that helps.

Reset8400 · November 18, 2025, 1:37pm

This allowed me to finish the migration, thank you!

system · December 2, 2025, 1:37pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog-datanode migration error Graylog Central (peer support) data-node	8	82	December 2, 2025
Opensearch to Datanode migration Graylog Central (peer support)	1	290	April 25, 2025
Still can't seem to perform a Datanode migration Graylog Central (peer support) data-node	15	253	June 19, 2025
Data Node Migration - Run directory compatibility check - fail Graylog Central (peer support) data-node	7	99	October 31, 2025
Be a part of Graylog Development - Call for testers! Development data-node	12	215	August 30, 2024

Data Node Migration Failure

Related topics