I tried to ping. Everyone ping every fqdn
It does not work better.
Is there anywhere else I have to change the name?
It really looks like this issue : Pipeline Stages with the same Priority
Could you send me example datanode.conf and server.conf files for a 3-node cluster setup
Hi @Etny,
I think you are on the right track after changing the node_name now. I guess the only problem now is that your data node keystore still only contains certs issued for the nodes without .local.lan
After removing this, you should check that in graylog.log that your Graylog server issues new ones for the correct subject.
Like this from above, only with the .local.lan: 2025-05-14T15:49:13.707+02:00 INFO [CaKeystore] Signing certificate for node e899cf63-a07f-45e4-99af-18f8716188e9, subject: CN=datanode-2
Hi @ matthias_gl
It seems to be good… but it does not :
2025-05-20T16:08:08.751+02:00 ERROR [DatanodeConnectivityCheck] Unable to retrieve version from indexer node:
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 1 attempts.
at com.github.rholder.retry.Retryer.call(Retryer.java:174) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.probe(VersionProbe.java:107) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.DatanodeConnectivityCheck.probe(DatanodeConnectivityCheck.java:52) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.verifyActualConnection(PreflightResource.java:117) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.getProvisioningState(PreflightResource.java:108) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.lambda$listDataNodes$0(PreflightResource.java:95) ~[graylog.jar:?]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.listDataNodes(PreflightResource.java:103) ~[graylog.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [graylog.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-05-20T16:08:10.347+02:00 INFO [CaKeystore] Signing certificate for node cfbabaf3-e873-4977-901c-14f13b302b11, subject: CN=datanode-3.local.lan
2025-05-20T16:08:10.384+02:00 INFO [CaKeystore] Signing certificate for node 4f15e49d-84d3-4dc7-9c13-49153bac64b9, subject: CN=datanode-1.local.lan
2025-05-20T16:08:10.419+02:00 INFO [CaKeystore] Signing certificate for node e899cf63-a07f-45e4-99af-18f8716188e9, subject: CN=datanode-2.local.lan
2025-05-20T16:13:47.262+02:00 INFO [CustomCAX509TrustManager] CA changed, refreshing trust manager
2025-05-20T16:13:58.076+02:00 INFO [PreflightJerseyService] Shutting down HTTP listener at <0.0.0.0:9000>
2025-05-20T16:13:58.077+02:00 INFO [Periodicals] Shutting down periodical [org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical].
2025-05-20T16:13:58.077+02:00 INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2025-05-20T16:13:58.077+02:00 INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
I deleted data and config on each datanode, launched preflight… I’m i at the same point than 12 days ago !
-Not all the node can works :
Could you please provide your opensearch.yml configuration files and maybe the logs of datanode-2 and datanode-3? There seems to be some problem for the nodes joining a cluster.
And, just to be sure. There are no other opensearch installations on the machines where you are running data node?
