Still can't seem to perform a Datanode migration

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
During the datanode migration wizard I am stuck on Step 3 3. Provision the Data Node’s certificate.

2. Describe your environment:

  • OS Information:
    Ubuntu 24.04
  • Package Version:
    6.2.2
  • Service logs, configurations, and environment variables:
2025-05-20T11:51:45.573-07:00 INFO  [MigrationStateMachineBuilder] Migration selected in menu, show welcome page
2025-05-20T11:51:45.707-07:00 ERROR [ChunkedBulkIndexer] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_503], id [795e7a80-35ab-11f0-bdce-005056ab0aa8], message [OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]
2025-05-20T11:51:54.044-07:00 WARN  [MaxmindDataAdapter] Unable to parse IP address, returning empty result.
2025-05-20T11:52:09.773-07:00 INFO  [CaKeystore] Signing certificate for  node ccd54941-7448-4ee6-9303-f75a9d49bbe8, subject: CN=gl3.REDACTED.org
2025-05-20T11:52:09.985-07:00 INFO  [CaKeystore] Signing certificate for  node 986646c7-4099-4afc-9868-6c5ca8e9c2d8, subject: CN=gl2.REDACTED.org
2025-05-20T11:52:10.057-07:00 INFO  [CaKeystore] Signing certificate for  node b9adf5c6-a0b9-409e-a762-cf8dd1295e88, subject: CN=gl1.REDACTED.org
2025-05-20T11:52:10.281-07:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: java.net.ConnectException: Failed to connect to gl2.REDACTED.org/10.50.2.202:8999
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:94) ~[graylog.jar:?]
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) ~[?:?]
	at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractTask.compute(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.runOnAllNodes(DatanodeRestApiProxy.java:89) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:121) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.request(DataNodeRestApiProxyResource.java:126) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.requestGet(DataNodeRestApiProxyResource.java:81) ~[graylog.jar:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [graylog.jar:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.ConnectException: Failed to connect to gl2.REDACTED.org/10.50.2.202:8999
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at org.graylog2.shared.bindings.providers.OkHttpClientProvider$UserAgentInterceptor.intercept(OkHttpClientProvider.java:78) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:147) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:91) ~[graylog.jar:?]
	... 45 more
	Suppressed: java.io.IOException: unexpected end of stream on https://gl2.REDACTED.org:8999/...
		at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:210) ~[graylog.jar:?]
		at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110) ~[graylog.jar:?]
		at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at org.graylog2.shared.bindings.providers.OkHttpClientProvider$UserAgentInterceptor.intercept(OkHttpClientProvider.java:78) ~[graylog.jar:?]
		at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
		at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[graylog.jar:?]
		at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
		at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:147) ~[graylog.jar:?]
		at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:91) ~[graylog.jar:?]
		at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source) ~[?:?]
		at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) ~[?:?]
		at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source) ~[?:?]
		at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
		at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
		at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
		at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(Unknown Source) ~[?:?]
		at java.base/java.util.stream.AbstractTask.compute(Unknown Source) ~[?:?]
		at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source) ~[?:?]
		at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
		at java.base/java.util.concurrent.ForkJoinTask.invoke(Unknown Source) ~[?:?]
		at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source) ~[?:?]
		at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
		at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
		at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.runOnAllNodes(DatanodeRestApiProxy.java:89) ~[graylog.jar:?]
		at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:121) ~[graylog.jar:?]
		at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.request(DataNodeRestApiProxyResource.java:126) ~[graylog.jar:?]
		at org.graylog2.rest.resources.datanodes.DataNodeRestApiProxyResource.requestGet(DataNodeRestApiProxyResource.java:81) ~[graylog.jar:?]
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
		at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
		at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
		at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) [graylog.jar:?]
		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
		at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
		at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
		at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
		at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266) [graylog.jar:?]
		at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253) [graylog.jar:?]
		at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696) [graylog.jar:?]
		at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [graylog.jar:?]
		at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [graylog.jar:?]
		at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [graylog.jar:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
		at java.base/java.lang.Thread.run(Unknown Source) [?:?]
	Caused by: java.io.EOFException: \n not found: limit=0 content=…
		at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335) ~[graylog.jar:?]
		at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) ~[graylog.jar:?]
		at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180) ~[graylog.jar:?]
		... 62 more
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source) ~[?:?]
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) ~[?:?]
	at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source) ~[?:?]
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
	at java.base/java.net.Socket.connect(Unknown Source) ~[?:?]
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[graylog.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[graylog.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at org.graylog2.shared.bindings.providers.OkHttpClientProvider$UserAgentInterceptor.intercept(OkHttpClientProvider.java:78) ~[graylog.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[graylog.jar:?]
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.request(DatanodeRestApiProxy.java:147) ~[graylog.jar:?]
	at org.graylog2.rest.resources.datanodes.DatanodeRestApiProxy.lambda$runOnAllNodes$0(DatanodeRestApiProxy.java:91) ~[graylog.jar:?]
	... 45 more

3. What steps have you already taken to try and solve the problem?

4. How can the community help?
I would like to know what REST resource is being accessed

ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: java.net.ConnectException: Failed to connect to gl2.REDACTED.org/10.50.2.202:8999

It seems like this call will randomly fail against any one of my 3 nodes.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hi @workinonit,
Could you please post your datanode logs? If you can’t connect, maybe it crashed for a reason and we’ll see that in its logs.

3 
2025-05-22T08:45:51.831-07:00 WARN  [CmdLineTool] Couldn't create native lib dir </native_libs>. Unable to set ZstdTempFolder system property.
java.nio.file.AccessDeniedException: /native_libs
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
2025-05-22T08:45:52.036-07:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-05-22T08:45:52.040-07:00 WARN  [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-05-22T08:45:52.541-07:00 INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-05-22T08:45:52.965-07:00 INFO  [cluster] Adding discovered server gl1.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.025-07:00 INFO  [cluster] Adding discovered server gl2.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.028-07:00 INFO  [cluster] Adding discovered server gl3.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.059-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-59-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED.org:27017, gl2.REDACTED.org:27017, gl3.REDACTED.org:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-22T08:45:53.062-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-59-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED.org:27017, gl2.REDACTED.org:27017, gl3.REDACTED.org:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-22T08:45:53.119-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl1.REDACTED.org:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=29922938, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl1.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000ad, setVersion=3, topologyVersion=TopologyVersion{processId=6815193d35320a90fef4e421, counter=6}, lastWriteDate=Thu May 22 08:45:53 PDT 2025, lastUpdateTimeNanos=1715583017146624}
2025-05-22T08:45:53.120-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl2.REDACTED.org:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=29935941, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl2.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=681518d85121fb5acb681180, counter=8}, lastWriteDate=Thu May 22 08:45:53 PDT 2025, lastUpdateTimeNanos=1715583017160251}
2025-05-22T08:45:53.120-07:00 INFO  [cluster] Waiting for server to become available for operation with ID 5. Remaining time: 29975 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=gl3.REDACTED.org:27017, type=UNKNOWN, state=CONNECTING}, {address=gl1.REDACTED.org:27017, type=UNKNOWN, state=CONNECTING}, {address=gl2.REDACTED.org:27017, type=UNKNOWN, state=CONNECTING}].
2025-05-22T08:45:53.121-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl3.REDACTED.org:27017, type=REPLICA_SET_OTHER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=28115794, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl3.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6815186290f681036e5123b4, counter=7}, lastWriteDate=Thu Apr 03 23:06:33 PDT 2025, lastUpdateTimeNanos=1715583015359325}
2025-05-22T08:45:53.134-07:00 INFO  [cluster] Discovered replica set primary gl1.REDACTED.org:27017 with max election id 7fffffff00000000000000ad and max set version 3
2025-05-22T08:45:53.265-07:00 INFO  [DatanodeDirectories] Opensearch of the node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/graylog/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-05-22T08:45:53.301-07:00 INFO  [cluster] Adding discovered server gl1.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.303-07:00 INFO  [cluster] Adding discovered server gl2.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.305-07:00 INFO  [cluster] Adding discovered server gl3.REDACTED.org:27017 to client view of cluster
2025-05-22T08:45:53.305-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl1.REDACTED.org:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=926532, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl1.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000ad, setVersion=3, topologyVersion=TopologyVersion{processId=6815193d35320a90fef4e421, counter=6}, lastWriteDate=Thu May 22 08:45:53 PDT 2025, lastUpdateTimeNanos=1715583227667598}
2025-05-22T08:45:53.306-07:00 INFO  [cluster] Discovered replica set primary gl1.REDACTED.org:27017 with max election id 7fffffff00000000000000ad and max set version 3
2025-05-22T08:45:53.307-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-59-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED.org:27017, gl2.REDACTED.org:27017, gl3.REDACTED.org:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-22T08:45:53.307-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl2.REDACTED.org:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1278893, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl2.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=681518d85121fb5acb681180, counter=8}, lastWriteDate=Thu May 22 08:45:53 PDT 2025, lastUpdateTimeNanos=1715583229548361}
2025-05-22T08:45:53.308-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-59-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED.org:27017, gl2.REDACTED.org:27017, gl3.REDACTED.org:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-22T08:45:53.309-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl3.REDACTED.org:27017, type=REPLICA_SET_OTHER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1318204, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl3.REDACTED.org:27017, hosts=[gl3.REDACTED.org:27017, gl2.REDACTED.org:27017, gl1.REDACTED.org:27017], passives=[], arbiters=[], primary='gl1.REDACTED.org:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6815186290f681036e5123b4, counter=7}, lastWriteDate=Thu Apr 03 23:06:33 PDT 2025, lastUpdateTimeNanos=1715583231402242}
2025-05-22T08:45:53.398-07:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.8
2025-05-22T08:45:53.403-07:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
5

Log continued…

2025-05-22T08:46:59.567-07:00 INFO  [OpensearchDataDirCompatibilityCheck] Found 321 indices and all of them are valid with current opensearch version 2.15.0
2025-05-22T08:46:59.963-07:00 INFO  [DatanodeDirectories] Opensearch of the node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/graylog/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-05-22T08:47:00.227-07:00 INFO  [DatanodeBootstrap] Graylog Data Node 6.2.2+13ba949 starting up (command: datanode)
2025-05-22T08:47:00.228-07:00 INFO  [DatanodeBootstrap] JRE: Eclipse Adoptium 17.0.15 on Linux 6.8.0-59-generic
2025-05-22T08:47:00.228-07:00 INFO  [DatanodeBootstrap] Deployment: deb
2025-05-22T08:47:00.229-07:00 INFO  [DatanodeBootstrap] OS: Ubuntu 24.04.2 LTS (noble)
2025-05-22T08:47:00.229-07:00 INFO  [DatanodeBootstrap] Arch: amd64
2025-05-22T08:47:00.288-07:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-05-22T08:47:00.288-07:00 INFO  [PeriodicalsService] Delaying start of 0 periodicals until this node becomes leader ...
2025-05-22T08:47:00.289-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-05-22T08:47:00.290-07:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-05-22T08:47:00.314-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-05-22T08:47:00.314-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-05-22T08:47:00.315-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-05-22T08:47:00.316-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s].
2025-05-22T08:47:00.325-07:00 INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-05-22T08:47:00.387-07:00 INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-05-22T08:47:00.394-07:00 INFO  [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
2025-05-22T08:47:00.395-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 10.50.2.201, gl1.REDACTED.org, ip6-localhost
2025-05-22T08:47:00.396-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following serial number: 1747773719964
2025-05-22T08:47:00.396-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following validity: 2025-05-20T13:41:59.000-0700 - 2075-05-08T13:41:59.000-0700
2025-05-22T08:47:00.397-07:00 INFO  [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
2025-05-22T08:47:00.397-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 10.50.2.201, gl1.REDACTED.org, ip6-localhost
2025-05-22T08:47:00.398-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following serial number: 1747773719964
2025-05-22T08:47:00.398-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following validity: 2025-05-20T13:41:59.000-0700 - 2075-05-08T13:41:59.000-0700
2025-05-22T08:47:00.401-07:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-05-22T08:47:00.407-07:00 INFO  [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=32, OpensearchConfigurationService [RUNNING]=117}
2025-05-22T08:47:00.407-07:00 INFO  [DatanodeBootstrap] Graylog DataNode datanode up and running.
2025-05-22T08:47:00.417-07:00 INFO  [JerseyService] Starting Data node REST API
2025-05-22T08:47:00.426-07:00 INFO  [OpensearchProcessService] OpenSearch starting up
2025-05-22T08:47:00.427-07:00 ERROR [OpensearchProcessService] Could not start up data node
org.graylog.datanode.bootstrap.preflight.DatanodeLockFileException: Failed to open channel to lock file /graylog/opensearch/data/datanode.lock
	at org.graylog.datanode.bootstrap.preflight.DatanodeDirectoriesLockfileCheck.checkDatanodeLock(DatanodeDirectoriesLockfileCheck.java:71) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.opensearch.OpensearchProcessService.onConfigurationChange(OpensearchProcessService.java:135) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.opensearch.OpensearchProcessService.onConfigurationChangeEvent(OpensearchProcessService.java:126) ~[graylog-datanode.jar:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:84) ~[guava-33.4.7-jre.jar:?]
	at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:141) ~[guava-33.4.7-jre.jar:?]
	at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:70) ~[guava-33.4.7-jre.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [metrics-core-4.2.30.jar:4.2.30]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.nio.file.AccessDeniedException: /graylog/opensearch/data/datanode.lock
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source) ~[?:?]
	at java.base/java.nio.channels.FileChannel.open(Unknown Source) ~[?:?]
	at java.base/java.nio.channels.FileChannel.open(Unknown Source) ~[?:?]
	at org.graylog.datanode.bootstrap.preflight.DatanodeDirectoriesLockfileCheck.checkDatanodeLock(DatanodeDirectoriesLockfileCheck.java:68) ~[graylog-datanode.jar:?]
	... 13 more
2025-05-22T08:47:01.103-07:00 INFO  [Version] HV000001: Hibernate Validator 8.0.2.Final
2025-05-22T08:47:01.456-07:00 INFO  [NetworkListener] Started listener bound to [10.50.2.201:8999]
2025-05-22T08:47:01.457-07:00 INFO  [HttpServer] [HttpServer] Started.
2025-05-22T08:47:01.457-07:00 INFO  [JerseyService] Started REST API at <10.50.2.201:8999>
6

I will add that the .lock file it complains about is not even located in the specified directory.

7

No one else is having this issue or one like it?

8

server.conf

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = REDACTED
root_username = admin
root_password_sha2 = REDACTED
root_email = "graylog@REDACTED"
root_timezone = America/Los_Angeles
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.50.2.201:9000
stream_aware_field_types=false
elasticsearch_hosts = http://10.50.2.201:9200,http://10.50.2.202:9200,http://10.50.2.203:9200
enabled_index_rotation_strategies = time
rotation_strategy = time
elasticsearch_max_time_per_index = 30d
elasticsearch_max_number_of_indices = 30
retention_strategy = delete
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = true
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 7
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://graylog:Graylog2Mongo@gl1.REDACTED:27017,gl2.REDACTED:27017,gl3.REDACTED:27017/graylog?replicaSet=graylog
mongodb_max_connections = 1000
transport_email_enabled = true
transport_email_hostname = 198.189.126.3
transport_email_port = 25
transport_email_use_auth = false
transport_email_from_email = graylog@REDACTED
transport_email_use_tls = false
transport_email_use_ssl = false
proxied_requests_thread_pool_size = 64

datanode.conf

node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = REDACTED
root_password_sha2 = REDACTED
mongodb_uri = mongodb://graylog:Graylog2Mongo@gl1.REDACTED:27017,gl2.REDACTED:27017,gl3.REDACTED:27017/graylog?replicaSet=graylog
bind_address = 10.50.2.201
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
#opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_data_location = /graylog/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 8g
9

From this line:

Caused by: java.nio.file.AccessDeniedException: /graylog/opensearch/data/datanode.lock

I assume that you have some permissions problems for your datanode user. The /graylog path seems quite unusual, so I’d check users and permissions. Then your datanode should start fine and graylog server will be able to connect to it.

10

Thank you for the tip. My java stacktrace reading skills need to improve. Previously I had added the graylog-datanode user to the opensearch group and the opensearch user to the graylog-datanode group. I had also set the directory permissions to 757 like the migration assistant stated. But this time I just went nuts on the directory permissions chmod -R 777 /graylog/opensearch. My first datanode started up… but the other two don’t want to come up. My Opensearch cluster is reporting red so I need to clear that up and I am sure the other two nodes will come up. At least I hope. Thanks for the help. I will report back.

11

So now I am regressing. I stop processing. Opensearch is running. Things are green. If I start datanode, opensearch goes red. If I stop opensearch and start datanode then the cluster doesn’t know what is going on. There seems to be a trick during the migration process where I am supposed to stop one process and start another or something? Basically nothing is working at this point in the migration.

12

You can’t let opensearch and datanodes run simultaneously. They are both locking the data directory and fail if somebody else is using it.

The migration process should guide you through stopping the processing, stopping your opensearch cluster and starting the datanode cluster, which will overtake the existing data.

It’s hard to guess what’s happening for you, after problems you mentioned above. On the other hand it’s good that the existing opensearch cluster still works and is green.

If you went successfully through the preflight with provisioning, I’d suggest:

  • Stop graylog server
  • Comment out elasticsearch_hosts in your server.conf
  • Stop all the opensearch nodes
  • Start all data nodes, each one running on the machine where opensearch previously used to, the data_dir pointing to the correct location.
  • Start the server

If it fails, collect all the datanode and server logs and upload them somewhere where I can access it and investigate.

13

Ok I:

  • Stopped graylog-server
  • Commented out the elasticsearch_hosts in the config files on all 3 of my nodes
  • Stopped all the opensearch nodes
  • Started all graylog-datanodes. Noticed the nodes would not start because of permissions so I chmod -R 777 /graylog/opensearch
  • The data location in my datanode.conf file remains the same opensearch_data_location = /graylog/opensearch/data
  • Started the 3 datanode nodes and they seem to be running
  • Started graylog-server. Server does not start.
  • graylog-server/server.log contains the following repeating error:
2025-06-04T15:13:27.537-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-06-04T15:13:27.538-07:00 INFO  [VersionProbe] Indexer is not available. Retry #1
2025-06-04T15:13:32.540-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-06-04T15:13:32.540-07:00 INFO  [VersionProbe] Indexer is not available. Retry #2
2025-06-04T15:13:37.542-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-06-04T15:13:37.542-07:00 INFO  [VersionProbe] Indexer is not available. Retry #3
2025-06-04T15:13:42.544-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-06-04T15:13:42.544-07:00 INFO  [VersionProbe] Indexer is not available. Retry #4
2025-06-04T15:13:47.546-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-06-04T15:13:47.546-07:00 INFO  [VersionProbe] Indexer is not available. Retry #5
  • If I comment back in the elasticsearch_hosts then I get the following error:
2025-06-04T15:24:52.583-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.201:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:52.593-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.202:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:52.607-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.203:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:52.608-07:00 INFO  [VersionProbe] Indexer is not available. Retry #1
2025-06-04T15:24:57.611-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.201:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:57.615-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.202:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:57.617-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.203:9200/... - \n not found: limit=0 content=….
2025-06-04T15:24:57.618-07:00 INFO  [VersionProbe] Indexer is not available. Retry #2
2025-06-04T15:25:02.621-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.201:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:02.623-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.202:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:02.626-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.203:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:02.626-07:00 INFO  [VersionProbe] Indexer is not available. Retry #3
2025-06-04T15:25:07.630-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.201:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:07.633-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.202:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:07.635-07:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: unexpected end of stream on http://10.50.2.203:9200/... - \n not found: limit=0 content=….
2025-06-04T15:25:07.636-07:00 INFO  [VersionProbe] Indexer is not available. Retry #4
14

What do you see in datanode logs?

15

With opensearch and graylog server stopped on the other two nodes and opensearch stopped on gl1 here is the datanode.log from gl1

2025-06-05T08:17:40.069-07:00 WARN  [CmdLineTool] Couldn't create native lib dir </native_libs>. Unable to set ZstdTempFolder system property.
java.nio.file.AccessDeniedException: /native_libs
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
2025-06-05T08:17:40.167-07:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-06-05T08:17:40.170-07:00 WARN  [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-06-05T08:17:40.483-07:00 INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-06-05T08:17:40.791-07:00 INFO  [cluster] Adding discovered server gl1.REDACTED:27017 to client view of cluster
2025-06-05T08:17:40.823-07:00 INFO  [cluster] Adding discovered server gl2.REDACTED:27017 to client view of cluster
2025-06-05T08:17:40.826-07:00 INFO  [cluster] Adding discovered server gl3.REDACTED:27017 to client view of cluster
2025-06-05T08:17:40.847-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-60-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED:27017, gl2.REDACTED:27017, gl3.REDACTED:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-05T08:17:40.849-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-60-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED:27017, gl2.REDACTED:27017, gl3.REDACTED:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-05T08:17:40.866-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl2.REDACTED:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18826429, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl2.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6840cafae46c570c18c1fe56, counter=4}, lastWriteDate=Thu Jun 05 08:17:32 PDT 2025, lastUpdateTimeNanos=60122754099406}
2025-06-05T08:17:40.866-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl1.REDACTED:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18822663, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl1.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000b0, setVersion=3, topologyVersion=TopologyVersion{processId=6840ca45d658482a9b1648df, counter=6}, lastWriteDate=Thu Jun 05 08:17:32 PDT 2025, lastUpdateTimeNanos=60122754099230}
2025-06-05T08:17:40.867-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl3.REDACTED:27017, type=REPLICA_SET_OTHER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18826783, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl3.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6840cafa67665b6b74c390cd, counter=5}, lastWriteDate=Thu Apr 03 23:06:33 PDT 2025, lastUpdateTimeNanos=60122754099550}
2025-06-05T08:17:40.872-07:00 INFO  [cluster] Discovered replica set primary gl1.REDACTED:27017 with max election id 7fffffff00000000000000b0 and max set version 3
2025-06-05T08:17:40.878-07:00 INFO  [cluster] Waiting for server to become available for operation with ID 5. Remaining time: 29993 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=gl3.REDACTED:27017, type=UNKNOWN, state=CONNECTING}, {address=gl1.REDACTED:27017, type=UNKNOWN, state=CONNECTING}, {address=gl2.REDACTED:27017, type=UNKNOWN, state=CONNECTING}].
2025-06-05T08:17:40.969-07:00 INFO  [DatanodeDirectories] Opensearch of the node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/graylog/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-06-05T08:17:41.003-07:00 INFO  [cluster] Adding discovered server gl1.REDACTED:27017 to client view of cluster
2025-06-05T08:17:41.005-07:00 INFO  [cluster] Adding discovered server gl2.REDACTED:27017 to client view of cluster
2025-06-05T08:17:41.007-07:00 INFO  [cluster] Adding discovered server gl3.REDACTED:27017 to client view of cluster
2025-06-05T08:17:41.007-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl1.REDACTED:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1037241, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl1.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000b0, setVersion=3, topologyVersion=TopologyVersion{processId=6840ca45d658482a9b1648df, counter=6}, lastWriteDate=Thu Jun 05 08:17:32 PDT 2025, lastUpdateTimeNanos=60122900969659}
2025-06-05T08:17:41.009-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl2.REDACTED:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1184598, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl2.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6840cafae46c570c18c1fe56, counter=4}, lastWriteDate=Thu Jun 05 08:17:32 PDT 2025, lastUpdateTimeNanos=60122902863858}
2025-06-05T08:17:41.009-07:00 INFO  [cluster] Discovered replica set primary gl1.REDACTED:27017 with max election id 7fffffff00000000000000b0 and max set version 3
2025-06-05T08:17:41.009-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-60-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED:27017, gl2.REDACTED:27017, gl3.REDACTED:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-05T08:17:41.011-07:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=gl3.REDACTED:27017, type=REPLICA_SET_OTHER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1086588, minRoundTripTimeNanos=0, setName='graylog', canonicalAddress=gl3.REDACTED:27017, hosts=[gl3.REDACTED:27017, gl2.REDACTED:27017, gl1.REDACTED:27017], passives=[], arbiters=[], primary='gl1.REDACTED:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, topologyVersion=TopologyVersion{processId=6840cafa67665b6b74c390cd, counter=5}, lastWriteDate=Thu Apr 03 23:06:33 PDT 2025, lastUpdateTimeNanos=60122904818032}
2025-06-05T08:17:41.011-07:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-60-generic"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@6bc24e72, com.mongodb.Jep395RecordCodecProvider@724aefc3, com.mongodb.KotlinCodecProvider@7cfb4736]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[gl1.REDACTED:27017, gl2.REDACTED:27017, gl3.REDACTED:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='graylog', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-05T08:17:41.061-07:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.8
2025-06-05T08:17:41.063-07:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-06-05T08:17:42.695-07:00 INFO  [OpensearchDataDirCompatibilityCheck] Found 171 indices and all of them are valid with current opensearch version 2.15.0
2025-06-05T08:17:43.001-07:00 INFO  [DatanodeDirectories] Opensearch of the node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/graylog/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-06-05T08:17:43.221-07:00 INFO  [DatanodeBootstrap] Graylog Data Node 6.2.2+13ba949 starting up (command: datanode)
2025-06-05T08:17:43.222-07:00 INFO  [DatanodeBootstrap] JRE: Eclipse Adoptium 17.0.15 on Linux 6.8.0-60-generic
2025-06-05T08:17:43.222-07:00 INFO  [DatanodeBootstrap] Deployment: deb
2025-06-05T08:17:43.222-07:00 INFO  [DatanodeBootstrap] OS: Ubuntu 24.04.2 LTS (noble)
2025-06-05T08:17:43.223-07:00 INFO  [DatanodeBootstrap] Arch: amd64
2025-06-05T08:17:43.254-07:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-06-05T08:17:43.255-07:00 INFO  [PeriodicalsService] Delaying start of 0 periodicals until this node becomes leader ...
2025-06-05T08:17:43.255-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s].
2025-06-05T08:17:43.304-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-06-05T08:17:43.305-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-06-05T08:17:43.305-07:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-06-05T08:17:43.306-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-06-05T08:17:43.307-07:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-06-05T08:17:43.308-07:00 INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-06-05T08:17:43.373-07:00 INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-06-05T08:17:43.380-07:00 INFO  [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
2025-06-05T08:17:43.382-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 10.50.2.201, gl1.REDACTED, ip6-localhost
2025-06-05T08:17:43.382-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following serial number: 1748884929871
2025-06-05T08:17:43.383-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch HTTP certificate has following validity: 2025-06-02T10:22:09.000-0700 - 2075-05-21T10:22:09.000-0700
2025-06-05T08:17:43.383-07:00 INFO  [TruststoreCreator] Adding key certificate chain of alias datanode to the truststore
2025-06-05T08:17:43.384-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following alternative names: 127.0.0.1, localhost, 0:0:0:0:0:0:0:1, 10.50.2.201, gl1.REDACTED, ip6-localhost
2025-06-05T08:17:43.384-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following serial number: 1748884929871
2025-06-05T08:17:43.384-07:00 INFO  [OpensearchSecurityConfigurationBean] Opensearch Transport certificate has following validity: 2025-06-02T10:22:09.000-0700 - 2075-05-21T10:22:09.000-0700
2025-06-05T08:17:43.387-07:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-06-05T08:17:43.391-07:00 INFO  [DatanodeBootstrap] Services started, startup times in ms: {OpensearchProcessService [RUNNING]=0, GracefulShutdownService [RUNNING]=0, PeriodicalsService [RUNNING]=55, OpensearchConfigurationService [RUNNING]=135}
2025-06-05T08:17:43.391-07:00 INFO  [DatanodeBootstrap] Graylog DataNode datanode up and running.
2025-06-05T08:17:43.409-07:00 INFO  [OpensearchProcessService] OpenSearch starting up
2025-06-05T08:17:43.411-07:00 INFO  [JerseyService] Starting Data node REST API
2025-06-05T08:17:43.412-07:00 INFO  [OpensearchCommandLineProcess] Creating opensearch keystore
2025-06-05T08:17:43.867-07:00 INFO  [Version] HV000001: Hibernate Validator 8.0.2.Final
2025-06-05T08:17:44.080-07:00 INFO  [NetworkListener] Started listener bound to [10.50.2.201:8999]
2025-06-05T08:17:44.082-07:00 INFO  [HttpServer] [HttpServer] Started.
2025-06-05T08:17:44.082-07:00 INFO  [JerseyService] Started REST API at <10.50.2.201:8999>
2025-06-05T08:17:44.375-07:00 INFO  [OpensearchCommandLineProcess] Created opensearch keystore in /var/lib/graylog-datanode/opensearch/config/opensearch5914833513061675846/opensearch.keystore
2025-06-05T08:17:48.191-07:00 INFO  [OpensearchCommandLineProcess] Added 4 keystore items
2025-06-05T08:17:48.280-07:00 INFO  [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/bin/opensearch
2025-06-05T08:17:48.397-07:00 INFO  [ClusterNodeStateTracer] Updating cluster node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 from UNCONFIGURED to STARTING (reason: PROCESS_STARTED)
2025-06-05T08:17:50.107-07:00 WARN  [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-05T08:17:50.108-07:00 WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/opensearch-2.15.0.jar)
2025-06-05T08:17:50.108-07:00 WARN  [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
2025-06-05T08:17:50.108-07:00 WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2025-06-05T08:17:50.647-07:00 WARN  [OpensearchProcessImpl] Jun 05, 2025 8:17:50 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
2025-06-05T08:17:50.648-07:00 WARN  [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
2025-06-05T08:17:50.977-07:00 WARN  [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2025-06-05T08:17:50.977-07:00 WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/lib/opensearch-2.15.0.jar)
2025-06-05T08:17:50.977-07:00 WARN  [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
2025-06-05T08:17:50.977-07:00 WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2025-06-05T08:17:50.987-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:50,986][INFO ][o.o.n.Node               ] [gl1.REDACTED] version[2.15.0], pid[23782], build[tar/61dbcd0795c9bfe9b81e5762175414bc38bbcadf/2024-06-20T03:26:49.193630411Z], OS[Linux/6.8.0-60-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
2025-06-05T08:17:50.988-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:50,988][INFO ][o.o.n.Node               ] [gl1.REDACTED] JVM home [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/jdk], using bundled JDK/JRE [true]
2025-06-05T08:17:50.988-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:50,988][INFO ][o.o.n.Node               ] [gl1.REDACTED] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-9870987797403149347, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Xms8g, -Xmx8g, -Dopensearch.transport.cname_in_publish_address=true, -Djavax.net.ssl.trustStore=datanode-truststore.p12, -Djavax.net.ssl.trustStorePassword=treqEwobMIRLsnxmWqVAvVKJXBXRgqxOQceazpYWoKiWkbArsYWCJtanTDfMcDkDrDVocwjuvlHfUWiPMdRWguaIwErNUzswiBDzVVEyQweMlmkxgyNIEKOYhPLUhREAtMGwWgzhrMAjwTLOGouDtmPjMZOBejJsGLVuZSlauhGBJPFaYRjWDvUMnmXKbpGmpoFhthZyOlIHjHsHGjkXuLkjgbpCoSDJaIFyXtABkBYeJVBtbIRUUHMjeqmPwmyU, -Djavax.net.ssl.trustStoreType=pkcs12, -XX:MaxDirectMemorySize=4294967296, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch5914833513061675846, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
2025-06-05T08:17:51.932-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:51,932][INFO ][o.o.s.s.t.SSLConfig      ] [gl1.REDACTED] SSL dual mode is disabled
16 
2025-06-05T08:17:51.933-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:51,932][INFO ][o.o.s.OpenSearchSecurityPlugin] [gl1.REDACTED] OpenSearch Config path is /var/lib/graylog-datanode/opensearch/config/opensearch5914833513061675846
2025-06-05T08:17:52.131-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,130][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] JVM supports TLSv1.3
2025-06-05T08:17:52.132-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,132][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] Config directory is /var/lib/graylog-datanode/opensearch/config/opensearch5914833513061675846/, from there the key- and truststore files are resolved relatively
2025-06-05T08:17:52.210-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,209][INFO ][o.o.s.s.u.SSLCertificateHelper] [gl1.REDACTED] No alias given, use the first one: datanode
2025-06-05T08:17:52.211-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,210][WARN ][o.o.s.s.u.SSLCertificateHelper] [gl1.REDACTED] Certificate chain for alias datanode contains a root certificate
2025-06-05T08:17:52.554-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,553][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] HTTPS client auth mode OPTIONAL
2025-06-05T08:17:52.569-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,569][INFO ][o.o.s.s.u.SSLCertificateHelper] [gl1.REDACTED] No alias given, use the first one: datanode
2025-06-05T08:17:52.570-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,570][WARN ][o.o.s.s.u.SSLCertificateHelper] [gl1.REDACTED] Certificate chain for alias datanode contains a root certificate
2025-06-05T08:17:52.610-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,610][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] TLS Transport Client Provider : JDK
2025-06-05T08:17:52.610-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,610][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] TLS Transport Server Provider : JDK
2025-06-05T08:17:52.610-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,610][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] TLS HTTP Provider             : JDK
2025-06-05T08:17:52.611-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,610][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
2025-06-05T08:17:52.611-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,611][INFO ][o.o.s.s.DefaultSecurityKeyStore] [gl1.REDACTED] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
2025-06-05T08:17:52.642-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:52,642][INFO ][o.o.s.OpenSearchSecurityPlugin] [gl1.REDACTED] Clustername: datanode-cluster
2025-06-05T08:17:53.148-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,148][INFO ][o.o.i.r.ReindexPlugin    ] [gl1.REDACTED] ReindexPlugin reloadSPI called
2025-06-05T08:17:53.149-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,149][INFO ][o.o.i.r.ReindexPlugin    ] [gl1.REDACTED] Unable to find any implementation for RemoteReindexExtension
2025-06-05T08:17:53.157-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,157][INFO ][o.o.j.JobSchedulerPlugin ] [gl1.REDACTED] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
2025-06-05T08:17:53.158-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,158][INFO ][o.o.j.JobSchedulerPlugin ] [gl1.REDACTED] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
2025-06-05T08:17:53.169-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,169][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [aggs-matrix-stats]
2025-06-05T08:17:53.170-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,170][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [analysis-common]
2025-06-05T08:17:53.170-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,170][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [cache-common]
2025-06-05T08:17:53.170-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,170][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [geo]
2025-06-05T08:17:53.171-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,170][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [ingest-common]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,170][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [ingest-geoip]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,171][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [ingest-user-agent]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,171][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [lang-expression]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,171][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [lang-mustache]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,171][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [lang-painless]
2025-06-05T08:17:53.172-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,171][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [mapper-extras]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,172][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [opensearch-dashboards]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,172][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [parent-join]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,172][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [percolator]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,172][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [rank-eval]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,172][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [reindex]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,173][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [repository-url]
2025-06-05T08:17:53.173-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,173][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [search-pipeline-common]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,173][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [systemd]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,173][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded module [transport-netty4]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,173][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-anomaly-detection]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,174][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-asynchronous-search]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,174][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-cross-cluster-replication]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,174][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-flow-framework]
2025-06-05T08:17:53.174-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,174][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-index-management]
2025-06-05T08:17:53.176-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,174][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-job-scheduler]
2025-06-05T08:17:53.177-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,175][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-ml]
2025-06-05T08:17:53.177-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,175][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-security]
2025-06-05T08:17:53.177-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,175][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [opensearch-skills]
2025-06-05T08:17:53.177-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,175][INFO ][o.o.p.PluginsService     ] [gl1.REDACTED] loaded plugin [repository-s3]
2025-06-05T08:17:53.189-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,189][INFO ][o.o.s.OpenSearchSecurityPlugin] [gl1.REDACTED] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
2025-06-05T08:17:53.191-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,191][INFO ][o.o.e.ExtensionsManager  ] [gl1.REDACTED] ExtensionsManager initialized
2025-06-05T08:17:53.205-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,205][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [gl1.REDACTED] Using MemorySegmentIndexInput with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2025-06-05T08:17:53.211-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,211][INFO ][o.o.e.NodeEnvironment    ] [gl1.REDACTED] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [661.4gb], net total_space [738gb], types [ext4]
2025-06-05T08:17:53.213-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,212][INFO ][o.o.e.NodeEnvironment    ] [gl1.REDACTED] heap size [8gb], compressed ordinary object pointers [true]
2025-06-05T08:17:53.374-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:53,374][INFO ][o.o.n.Node               ] [gl1.REDACTED] node name [gl1.REDACTED], node ID [g8NKTxZoS_WRmVQTzQOfyg], cluster name [datanode-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
2025-06-05T08:17:53.397-07:00 WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 23782 unavailable. Cause: Connection refused
2025-06-05T08:17:55.524-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,524][WARN ][o.o.s.c.Salt             ] [gl1.REDACTED] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
2025-06-05T08:17:55.549-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,549][ERROR][o.o.s.a.s.SinkProvider   ] [gl1.REDACTED] Default endpoint could not be created, auditlog will not work properly.
2025-06-05T08:17:55.551-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,550][WARN ][o.o.s.a.r.AuditMessageRouter] [gl1.REDACTED] No default storage available, audit log may not work properly. Please check configuration.
2025-06-05T08:17:55.552-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,550][INFO ][o.o.s.a.i.AuditLogImpl   ] [gl1.REDACTED] Message routing enabled: false
2025-06-05T08:17:55.582-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,582][INFO ][o.o.s.f.SecurityFilter   ] [gl1.REDACTED] <NONE> indices are made immutable.
2025-06-05T08:17:55.708-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,708][INFO ][o.o.m.b.MLCircuitBreakerService] [gl1.REDACTED] Registered ML memory breaker.
2025-06-05T08:17:55.709-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,708][INFO ][o.o.m.b.MLCircuitBreakerService] [gl1.REDACTED] Registered ML disk breaker.
2025-06-05T08:17:55.709-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,709][INFO ][o.o.m.b.MLCircuitBreakerService] [gl1.REDACTED] Registered ML native memory breaker.
2025-06-05T08:17:55.791-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,791][INFO ][o.r.Reflections          ] [gl1.REDACTED] Reflections took 41 ms to scan 1 urls, producing 26 keys and 67 values 
2025-06-05T08:17:55.942-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:55,942][INFO ][o.o.t.b.CircuitBreakerService] [gl1.REDACTED] Registered memory breaker.
2025-06-05T08:17:56.167-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,167][WARN ][stderr                   ] [gl1.REDACTED] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
2025-06-05T08:17:56.167-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,167][WARN ][stderr                   ] [gl1.REDACTED] SLF4J: Defaulting to no-operation (NOP) logger implementation
2025-06-05T08:17:56.167-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,167][WARN ][stderr                   ] [gl1.REDACTED] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-06-05T08:17:56.477-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,477][INFO ][o.o.t.NettyAllocator     ] [gl1.REDACTED] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
2025-06-05T08:17:56.481-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,481][INFO ][o.o.s.s.t.SSLConfig      ] [gl1.REDACTED] SSL dual mode is disabled
2025-06-05T08:17:56.571-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,571][INFO ][o.o.d.DiscoveryModule    ] [gl1.REDACTED] using discovery type [zen] and seed hosts providers [settings, file]
2025-06-05T08:17:56.956-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:56,956][WARN ][o.o.g.DanglingIndicesState] [gl1.REDACTED] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2025-06-05T08:17:57.430-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,430][INFO ][o.o.n.Node               ] [gl1.REDACTED] initialized
2025-06-05T08:17:57.430-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,430][INFO ][o.o.n.Node               ] [gl1.REDACTED] starting ...
2025-06-05T08:17:57.505-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,505][INFO ][o.o.t.TransportService   ] [gl1.REDACTED] publish_address {gl1.REDACTED/10.50.2.201:9300}, bound_addresses {10.50.2.201:9300}
2025-06-05T08:17:57.507-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,506][INFO ][o.o.t.TransportService   ] [gl1.REDACTED] Remote clusters initialized successfully.
2025-06-05T08:17:57.907-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,906][INFO ][o.o.b.BootstrapChecks    ] [gl1.REDACTED] bound or publishing to a non-loopback address, enforcing bootstrap checks
2025-06-05T08:17:57.911-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:17:57,911][INFO ][o.o.c.c.Coordinator      ] [gl1.REDACTED] cluster UUID [x1-UwPJURjywfdoAewBqTA]
2025-06-05T08:18:03.311-07:00 WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 23782 unavailable. Cause: Connection refused
2025-06-05T08:18:07.921-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:07,920][WARN ][o.o.c.c.ClusterFormationFailureHelper] [gl1.REDACTED] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [g8NKTxZoS_WRmVQTzQOfyg, 773ThGqmQrSv9pTGqka66Q, ii8R7YFiSRG-4NsiMw6vtQ], have discovered [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [127.0.0.1:9300, [::1]:9300] from hosts providers and [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 184, last-accepted version 658821 in term 184
2025-06-05T08:18:13.308-07:00 WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 23782 unavailable. Cause: Connection refused
2025-06-05T08:18:17.922-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:17,922][WARN ][o.o.c.c.ClusterFormationFailureHelper] [gl1.REDACTED] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [g8NKTxZoS_WRmVQTzQOfyg, 773ThGqmQrSv9pTGqka66Q, ii8R7YFiSRG-4NsiMw6vtQ], have discovered [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [127.0.0.1:9300, [::1]:9300] from hosts providers and [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 184, last-accepted version 658821 in term 184
2025-06-05T08:18:23.308-07:00 WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 23782 unavailable. Cause: Connection refused
2025-06-05T08:18:27.922-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,921][WARN ][o.o.n.Node               ] [gl1.REDACTED] timed out while waiting for initial discovery state - timeout: 30s
2025-06-05T08:18:27.924-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,923][WARN ][o.o.c.c.ClusterFormationFailureHelper] [gl1.REDACTED] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [g8NKTxZoS_WRmVQTzQOfyg, 773ThGqmQrSv9pTGqka66Q, ii8R7YFiSRG-4NsiMw6vtQ], have discovered [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [127.0.0.1:9300, [::1]:9300] from hosts providers and [{gl1.REDACTED}{g8NKTxZoS_WRmVQTzQOfyg}{-xIA3K7ART2uSWq4eEhasw}{gl1.REDACTED}{10.50.2.201:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 184, last-accepted version 658821 in term 184
2025-06-05T08:18:27.929-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,929][INFO ][o.o.h.AbstractHttpServerTransport] [gl1.REDACTED] publish_address {gl1.REDACTED/10.50.2.201:9200}, bound_addresses {10.50.2.201:9200}
2025-06-05T08:18:27.929-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,929][INFO ][o.o.n.Node               ] [gl1.REDACTED] started
2025-06-05T08:18:27.930-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,930][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Will attempt to create index .opendistro_security and default configs if they are absent
2025-06-05T08:18:27.931-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,931][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Background init thread started. Install default config?: true
2025-06-05T08:18:27.931-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,931][INFO ][o.o.s.OpenSearchSecurityPlugin] [gl1.REDACTED] 0 OpenSearch Security modules loaded so far: []
2025-06-05T08:18:27.931-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:27,931][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:28.932-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:28,931][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:29.932-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:29,932][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:30.933-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:30,932][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:31.933-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:31,933][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:32.934-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:32,933][INFO ][o.o.s.c.ConfigurationRepository] [gl1.REDACTED] Wait for cluster to be available ...
2025-06-05T08:18:33.517-07:00 INFO  [OpensearchProcessImpl] [2025-06-05T08:18:33,516][ERROR][o.o.s.a.BackendRegistry  ] [gl1.REDACTED] Not yet initialized (you may need to run securityadmin)
2025-06-05T08:18:33.556-07:00 INFO  [ClusterNodeStateTracer] Updating cluster node b9adf5c6-a0b9-409e-a762-cf8dd1295e88 from STARTING to UNAVAILABLE (reason: HEALTH_CHECK_FAILED)
2025-06-05T08:18:33.562-07:00 WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 23782 unavailable. Cause: Unable to parse response body