Datanode showing Unavailable

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident: I recently built out a multi node graylog instance. I initially built this one VMware workstation. However, I had to convert the hard disks to hyper-v to migrate them to a different environment. Once I migrated everything, I was able to open up the UI, see both my graylog servers, all my mongoDBs were healthy, but my 3 datanodes are reporting unavailable.

2. Describe your environment:

  • OS Information:Ubuntu 24.04.4 LTS

  • Package Version:6.0.14-1

  • Service logs, configurations, and environment variables: 2 load balencers , 2 graylog servers , 3 mongoDBs, 3 Datanodes. All on same subnet.

3. What steps have you already taken to try and solve the problem?

I have restarted services, tried to re provision the datanodes, deleted things from mongoDB to switch them to “unconfigured”, changed to just using 1 datanode. And more/

4. How can the community help?

I need to find out why this broke when it moved to hyper-v. It was fully working on vmware.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @chilla67 ,

I guess you should see some errors in your datanode logs. Can you post some here, so I can suggest what to do next and what could be the cause of your problems? Thanks!

3

Attached is a picture of the datanode.log.

Let me know what else you would like to see.

(attachments)

Screenshot 2026-04-20 093156.jpg
Screenshot 2026-04-20 093156.jpg2315×1167 595 KB

4

Thank you, but the screenshot alone doesn’t show the actual cause, only that the datanode (management layer) can’t connect to the underlying opensearch process. The actual reason will be higher in the logs. Best would be to restart the datanode service and observe the first minute or two of logs. There will be probably some log lines from opensearch, telling you what’s causing issues.

5

Here is a log split into a few replies:

administrator@INDOT-DN01:~$ tail -f /var/log/graylog-datanode/datanode.log
2026-04-22T09:44:42.462-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3434 unavailable. Cause: Connection refused
2026-04-22T09:44:52.462-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3434 unavailable. Cause: Connection refused
2026-04-22T09:45:02.462-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 3434 unavailable. Cause: Connection refused
2026-04-22T09:45:03.878-04:00 INFO [Server] SIGNAL received. Shutting down.
2026-04-22T09:45:03.879-04:00 INFO [GracefulShutdown] Graceful shutdown initiated.
2026-04-22T09:45:03.881-04:00 INFO [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
2026-04-22T09:45:03.881-04:00 INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2026-04-22T09:45:03.881-04:00 INFO [GracefulShutdown] Goodbye.
2026-04-22T09:45:03.882-04:00 INFO [ClusterNodeStateTracer] Updating cluster node 65e90c32-e9b6-4128-afe0-5e67dbf36fc2 from UNAVAILABLE to UNAVAILABLE (reason: PROCESS_STOPPED)
2026-04-22T09:45:11.815-04:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2026-04-22T09:45:12.413-04:00 INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2026-04-22T09:45:12.624-04:00 INFO [cluster] Adding discovered server indot-mongo01:27017 to client view of cluster
2026-04-22T09:45:12.680-04:00 INFO [cluster] Adding discovered server indot-mongo02:27017 to client view of cluster
2026-04-22T09:45:12.713-04:00 INFO [cluster] Adding discovered server indot-mongo03:27017 to client view of cluster
2026-04-22T09:45:12.804-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo01:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=71883624, setName=‘rs0’, canonicalAddress=INDOT-MONGO01:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000003, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0eb3fa687e6d5a997bf6d, counter=5}, lastWriteDate=Wed Apr 22 09:45:12 EDT 2026, lastUpdateTimeNanos=518866016112419}
2026-04-22T09:45:12.806-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo03:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=32738672, setName=‘rs0’, canonicalAddress=INDOT-MONGO03:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=null, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0ec308644def10abfbe59, counter=3}, lastWriteDate=Wed Apr 22 09:45:12 EDT 2026, lastUpdateTimeNanos=518866009126735}
2026-04-22T09:45:12.809-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo02:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=41811066, setName=‘rs0’, canonicalAddress=INDOT-MONGO02:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=null, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0eb8d99e0da191c59c1da, counter=3}, lastWriteDate=Wed Apr 22 09:45:12 EDT 2026, lastUpdateTimeNanos=518866005909624}
2026-04-22T09:45:12.819-04:00 INFO [cluster] Discovered replica set primary indot-mongo01:27017 with max election id 7fffffff0000000000000003 and max set version 85770
2026-04-22T09:45:12.839-04:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.0.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.17.0-14-generic”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@4c18621b, com.mongodb.Jep395RecordCodecProvider@39c385d6, com.mongodb.KotlinCodecProvider@1cec219f]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[indot-mongo01:27017, indot-mongo02:27017, indot-mongo03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2026-04-22T09:45:12.844-04:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.0.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.17.0-14-generic”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@4c18621b, com.mongodb.Jep395RecordCodecProvider@39c385d6, com.mongodb.KotlinCodecProvider@1cec219f]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[indot-mongo01:27017, indot-mongo02:27017, indot-mongo03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2026-04-22T09:45:12.997-04:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 8.0.19
2026-04-22T09:45:13.125-04:00 INFO [DatanodeDirectories] Opensearch of the node 65e90c32-e9b6-4128-afe0-5e67dbf36fc2 uses following directories as its storage: DatanodeDirectories{dataTargetDir=‘/var/lib/graylog-datanode/opensearch/data’, logsTargetDir=‘/var/log/graylog-datanode/opensearch’, configurationSourceDir=‘Optional[/etc/graylog/datanode]’, configurationTargetDir=‘/var/lib/graylog-datanode/opensearch/config’, opensearchProcessConfigurationDir=‘/var/lib/graylog-datanode/opensearch/config/opensearch’}
2026-04-22T09:45:13.230-04:00 INFO [OpensearchConfigSync] Directory used for Opensearch process configuration is /var/lib/graylog-datanode/opensearch/config/opensearch
2026-04-22T09:45:13.273-04:00 INFO [OpensearchConfigSync] Synchronizing Opensearch configuration
2026-04-22T09:45:13.281-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12
2026-04-22T09:45:13.282-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2026-04-22T09:45:13.283-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2026-04-22T09:45:13.284-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch.yml
2026-04-22T09:45:13.285-04:00 INFO [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2026-04-22T09:45:13.313-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2026-04-22T09:45:13.316-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2026-04-22T09:45:13.316-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2026-04-22T09:45:13.318-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2026-04-22T09:45:13.321-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2026-04-22T09:45:13.321-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2026-04-22T09:45:13.321-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2026-04-22T09:45:13.322-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2026-04-22T09:45:13.324-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2026-04-22T09:45:13.324-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2026-04-22T09:45:13.325-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2026-04-22T09:45:13.328-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12
2026-04-22T09:45:13.328-04:00 INFO [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2026-04-22T09:45:13.330-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2026-04-22T09:45:13.330-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch.keystore
2026-04-22T09:45:13.330-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/http-keystore.p12
2026-04-22T09:45:13.332-04:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/unicast_hosts.txt
2026-04-22T09:45:13.340-04:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch
2026-04-22T09:45:13.343-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2026-04-22T09:45:13.345-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2026-04-22T09:45:13.348-04:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2026-04-22T09:45:13.349-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2026-04-22T09:45:13.352-04:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2026-04-22T09:45:13.357-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2026-04-22T09:45:13.359-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2026-04-22T09:45:13.360-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2026-04-22T09:45:13.362-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2026-04-22T09:45:13.367-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2026-04-22T09:45:13.368-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2026-04-22T09:45:13.369-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2026-04-22T09:45:13.371-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2026-04-22T09:45:13.372-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2026-04-22T09:45:13.374-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2026-04-22T09:45:13.375-04:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2026-04-22T09:45:13.380-04:00 INFO [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64]
2026-04-22T09:45:13.752-04:00 INFO [DatanodeDirectories] Opensearch of the node 65e90c32-e9b6-4128-afe0-5e67dbf36fc2 uses following directories as its storage: DatanodeDirectories{dataTargetDir=‘/var/lib/graylog-datanode/opensearch/data’, logsTargetDir=‘/var/log/graylog-datanode/opensearch’, configurationSourceDir=‘Optional[/etc/graylog/datanode]’, configurationTargetDir=‘/var/lib/graylog-datanode/opensearch/config’, opensearchProcessConfigurationDir=‘/var/lib/graylog-datanode/opensearch/config/opensearch’}
2026-04-22T09:45:13.884-04:00 INFO [cluster] Adding discovered server indot-mongo01:27017 to client view of cluster
2026-04-22T09:45:13.894-04:00 INFO [cluster] Adding discovered server indot-mongo02:27017 to client view of cluster
2026-04-22T09:45:13.902-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo01:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3300606, setName=‘rs0’, canonicalAddress=INDOT-MONGO01:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000003, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0eb3fa687e6d5a997bf6d, counter=5}, lastWriteDate=Wed Apr 22 09:45:13 EDT 2026, lastUpdateTimeNanos=518867117827287}
2026-04-22T09:45:13.905-04:00 INFO [cluster] Adding discovered server indot-mongo03:27017 to client view of cluster
2026-04-22T09:45:13.910-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo02:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2336958, setName=‘rs0’, canonicalAddress=INDOT-MONGO02:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=null, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0eb8d99e0da191c59c1da, counter=3}, lastWriteDate=Wed Apr 22 09:45:13 EDT 2026, lastUpdateTimeNanos=518867129217363}
2026-04-22T09:45:13.916-04:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.0.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.17.0-14-generic”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@4c18621b, com.mongodb.Jep395RecordCodecProvider@39c385d6, com.mongodb.KotlinCodecProvider@1cec219f]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[indot-mongo01:27017, indot-mongo02:27017, indot-mongo03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2026-04-22T09:45:13.921-04:00 INFO [cluster] Discovered replica set primary indot-mongo01:27017 with max election id 7fffffff0000000000000003 and max set version 85770
2026-04-22T09:45:13.924-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=indot-mongo03:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3634044, setName=‘rs0’, canonicalAddress=INDOT-MONGO03:27017, hosts=[INDOT-MONGO03:27017, INDOT-MONGO01:27017, INDOT-MONGO02:27017], passives=, arbiters=, primary=‘INDOT-MONGO01:27017’, tagSet=TagSet{}, electionId=null, setVersion=85770, topologyVersion=TopologyVersion{processId=69e0ec308644def10abfbe59, counter=3}, lastWriteDate=Wed Apr 22 09:45:13 EDT 2026, lastUpdateTimeNanos=518867143654517}
2026-04-22T09:45:13.930-04:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.0.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.17.0-14-generic”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@4c18621b, com.mongodb.Jep395RecordCodecProvider@39c385d6, com.mongodb.KotlinCodecProvider@1cec219f]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[indot-mongo01:27017, indot-mongo02:27017, indot-mongo03:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2026-04-22T09:45:16.377-04:00 INFO [DbEntitiesScanner] 14 entities have been scanned and added to DB Entity Catalog, it took 1.791 s
2026-04-22T09:45:16.443-04:00 INFO [ServerBootstrap] Graylog datanode 6.0.14+508aa86 starting up
2026-04-22T09:45:16.448-04:00 INFO [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.17.0-14-generic
2026-04-22T09:45:16.452-04:00 INFO [ServerBootstrap] Deployment: deb
2026-04-22T09:45:16.455-04:00 INFO [ServerBootstrap] OS: Ubuntu 24.04.4 LTS (noble)
2026-04-22T09:45:16.455-04:00 INFO [ServerBootstrap] Arch: amd64
2026-04-22T09:45:16.569-04:00 INFO [PeriodicalsService] Starting 6 periodicals …
2026-04-22T09:45:16.570-04:00 INFO [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader …
2026-04-22T09:45:16.571-04:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.ClusterManagerDiscovery] periodical in [0s], polling every [10s].
2026-04-22T09:45:16.604-04:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2026-04-22T09:45:16.609-04:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2026-04-22T09:45:16.610-04:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2026-04-22T09:45:16.613-04:00 INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2026-04-22T09:45:16.616-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2026-04-22T09:45:16.813-04:00 INFO [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, INDOT-DN01, ip6-localhost, 10.10.40.7
2026-04-22T09:45:16.828-04:00 INFO [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, INDOT-DN01, ip6-localhost, 10.10.40.7
2026-04-22T09:45:16.862-04:00 INFO [TruststoreCreator] Adding certificate transport-chain-CA-root to the truststore
2026-04-22T09:45:16.863-04:00 INFO [TruststoreCreator] Adding certificate http-chain-CA-root to the truststore
2026-04-22T09:45:16.994-04:00 INFO [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64]
2026-04-22T09:45:17.100-04:00 INFO [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, INDOT-DN01, ip6-localhost, 10.10.40.7
2026-04-22T09:45:17.111-04:00 INFO [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, INDOT-DN01, ip6-localhost, 10.10.40.7
2026-04-22T09:45:17.140-04:00 INFO [TruststoreCreator] Adding certificate transport-chain-CA-root to the truststore
2026-04-22T09:45:17.141-04:00 INFO [TruststoreCreator] Adding certificate http-chain-CA-root to the truststore
2026-04-22T09:45:17.244-04:00 INFO [OpensearchCommandLineProcess] No S3 repository configuration provided, skipping plugin initialization
2026-04-22T09:45:17.516-04:00 INFO [CommandLineProcess] Running process from /usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64/bin/opensearch
2026-04-22T09:45:17.529-04:00 INFO [JerseyService] Starting Data node REST API
2026-04-22T09:45:17.534-04:00 INFO [ClusterNodeStateTracer] Updating cluster node 65e90c32-e9b6-4128-afe0-5e67dbf36fc2 from UNCONFIGURED to STARTING (reason: PROCESS_STARTED)
2026-04-22T09:45:17.584-04:00 INFO [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, PeriodicalsService [RUNNING]=56, OpensearchProcessService [RUNNING]=1016}
2026-04-22T09:45:17.585-04:00 INFO [ServerBootstrap] Graylog DataNode datanode up and running.
2026-04-22T09:45:18.545-04:00 INFO [Version] HV000001: Hibernate Validator 8.0.1.Final
2026-04-22T09:45:19.107-04:00 INFO [NetworkListener] Started listener bound to [10.10.40.7:8999]
2026-04-22T09:45:19.111-04:00 INFO [HttpServer] [HttpServer] Started.
2026-04-22T09:45:19.112-04:00 INFO [JerseyService] Started REST API at <10.10.40.7:8999>
2026-04-22T09:45:20.720-04:00 WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2026-04-22T09:45:20.723-04:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64/lib/opensearch-2.12.0.jar)
2026-04-22T09:45:20.726-04:00 WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
2026-04-22T09:45:20.729-04:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2026-04-22T09:45:21.742-04:00 WARN [OpensearchProcessImpl] Apr 22, 2026 9:45:21 AM sun.util.locale.provider.LocaleProviderAdapter
2026-04-22T09:45:21.744-04:00 WARN [OpensearchProcessImpl] WARNING: COMPAT locale provider will be removed in a future release
2026-04-22T09:45:22.342-04:00 WARN [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2026-04-22T09:45:22.345-04:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64/lib/opensearch-2.12.0.jar)
2026-04-22T09:45:22.345-04:00 WARN [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
2026-04-22T09:45:22.345-04:00 WARN [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2026-04-22T09:45:22.374-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:22,370][INFO ][o.o.n.Node ] [INDOT-DN01] version[2.12.0], pid[38424], build[tar/2c355ce1a427e4a528778d4054436b5c4b756221/2024-02-20T02:18:49.874618333Z], OS[Linux/6.17.0-14-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-LTS]
2026-04-22T09:45:22.378-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:22,377][INFO ][o.o.n.Node ] [INDOT-DN01] JVM home [/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64/jdk], using bundled JDK/JRE [true]
2026-04-22T09:45:22.379-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:22,377][INFO ][o.o.n.Node ] [INDOT-DN01] JVM arguments [-Xshar

6

Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11169756531597373074, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.12.0-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/opensearch, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
2026-04-22T09:45:24.187-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,187][INFO ][o.o.s.s.t.SSLConfig ] [INDOT-DN01] SSL dual mode is disabled
2026-04-22T09:45:24.188-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,187][INFO ][o.o.s.OpenSearchSecurityPlugin] [INDOT-DN01] OpenSearch Config path is /var/lib/graylog-datanode/opensearch/config/opensearch
2026-04-22T09:45:24.584-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,583][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] JVM supports TLSv1.3
2026-04-22T09:45:24.586-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,586][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] Config directory is /var/lib/graylog-datanode/opensearch/config/opensearch/, from there the key- and truststore files are resolved relatively
2026-04-22T09:45:24.591-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,590][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.transport.keystore_password] has a secure counterpart [plugins.security.ssl.transport.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:24.598-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,598][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.transport.truststore_password] has a secure counterpart [plugins.security.ssl.transport.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:24.815-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:24,814][WARN ][o.o.s.s.u.SSLCertificateHelper] [INDOT-DN01] Certificate chain for alias datanode contains a root certificate
2026-04-22T09:45:25.179-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,178][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.http.keystore_password] has a secure counterpart [plugins.security.ssl.http.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:25.181-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,181][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] HTTPS client auth mode OPTIONAL
2026-04-22T09:45:25.210-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,210][WARN ][o.o.s.s.u.SSLCertificateHelper] [INDOT-DN01] Certificate chain for alias datanode contains a root certificate
2026-04-22T09:45:25.224-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,224][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.http.truststore_password] has a secure counterpart [plugins.security.ssl.http.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:25.294-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,293][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] TLS Transport Client Provider : JDK
2026-04-22T09:45:25.298-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,296][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] TLS Transport Server Provider : JDK
2026-04-22T09:45:25.299-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,296][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] TLS HTTP Provider : JDK
2026-04-22T09:45:25.300-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,299][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
2026-04-22T09:45:25.301-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,300][INFO ][o.o.s.s.DefaultSecurityKeyStore] [INDOT-DN01] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
2026-04-22T09:45:25.583-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:25,581][INFO ][o.o.s.OpenSearchSecurityPlugin] [INDOT-DN01] Clustername: datanode-cluster
2026-04-22T09:45:26.341-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,341][INFO ][o.o.i.r.ReindexPlugin ] [INDOT-DN01] ReindexPlugin reloadSPI called
2026-04-22T09:45:26.349-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,347][INFO ][o.o.i.r.ReindexPlugin ] [INDOT-DN01] Unable to find any implementation for RemoteReindexExtension
2026-04-22T09:45:26.395-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,393][INFO ][o.o.j.JobSchedulerPlugin ] [INDOT-DN01] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
2026-04-22T09:45:26.399-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,399][INFO ][o.o.j.JobSchedulerPlugin ] [INDOT-DN01] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
2026-04-22T09:45:26.424-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,424][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [aggs-matrix-stats]
2026-04-22T09:45:26.426-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,425][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [analysis-common]
2026-04-22T09:45:26.426-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,425][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [geo]
2026-04-22T09:45:26.426-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,425][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [ingest-common]
2026-04-22T09:45:26.428-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,428][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [ingest-geoip]
2026-04-22T09:45:26.429-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,428][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [ingest-user-agent]
2026-04-22T09:45:26.429-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,428][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [lang-expression]
2026-04-22T09:45:26.431-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,429][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [lang-mustache]
2026-04-22T09:45:26.431-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,430][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [lang-painless]
2026-04-22T09:45:26.433-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,430][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [mapper-extras]
2026-04-22T09:45:26.434-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,430][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [opensearch-dashboards]
2026-04-22T09:45:26.434-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,433][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [parent-join]
2026-04-22T09:45:26.434-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,433][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [percolator]
2026-04-22T09:45:26.434-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,433][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [rank-eval]
2026-04-22T09:45:26.435-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,433][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [reindex]
2026-04-22T09:45:26.437-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,436][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [repository-url]
2026-04-22T09:45:26.438-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,436][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [search-pipeline-common]
2026-04-22T09:45:26.438-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,436][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [systemd]
2026-04-22T09:45:26.438-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,436][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded module [transport-netty4]
2026-04-22T09:45:26.439-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,439][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-anomaly-detection]
2026-04-22T09:45:26.441-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,441][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-asynchronous-search]
2026-04-22T09:45:26.442-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,441][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-cross-cluster-replication]
2026-04-22T09:45:26.442-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,441][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-flow-framework]
2026-04-22T09:45:26.442-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,441][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-index-management]
2026-04-22T09:45:26.443-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,443][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-job-scheduler]
2026-04-22T09:45:26.444-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,443][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-ml]
2026-04-22T09:45:26.445-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,444][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-observability]
2026-04-22T09:45:26.445-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,444][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-security]
2026-04-22T09:45:26.445-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,444][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [opensearch-skills]
2026-04-22T09:45:26.445-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,444][INFO ][o.o.p.PluginsService ] [INDOT-DN01] loaded plugin [repository-s3]
2026-04-22T09:45:26.475-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,474][INFO ][o.o.s.OpenSearchSecurityPlugin] [INDOT-DN01] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting ‘http.compression: true’ in opensearch.yml
2026-04-22T09:45:26.491-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,490][INFO ][o.o.e.ExtensionsManager ] [INDOT-DN01] ExtensionsManager initialized
2026-04-22T09:45:26.521-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,521][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [INDOT-DN01] Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2026-04-22T09:45:26.532-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,531][INFO ][o.o.e.NodeEnvironment ] [INDOT-DN01] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [221.4gb], net total_space [245gb], types [ext4]
2026-04-22T09:45:26.533-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,532][INFO ][o.o.e.NodeEnvironment ] [INDOT-DN01] heap size [1gb], compressed ordinary object pointers [true]
2026-04-22T09:45:26.755-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:26,755][INFO ][o.o.n.Node ] [INDOT-DN01] node name [INDOT-DN01], node ID [UCR7rRE6QbCLt1fgj6DQgg], cluster name [datanode-cluster], roles [ingest, remote_cluster_client, data, cluster_manager, search]
2026-04-22T09:45:26.913-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:45:29.478-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:29,475][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.http.truststore_password] has a secure counterpart [plugins.security.ssl.http.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:29.482-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:29,477][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.transport.truststore_password] has a secure counterpart [plugins.security.ssl.transport.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:29.482-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:29,478][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.transport.keystore_password] has a secure counterpart [plugins.security.ssl.transport.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:29.488-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:29,480][WARN ][o.o.s.s.SecureSSLSettings] [INDOT-DN01] Setting [plugins.security.ssl.http.keystore_password] has a secure counterpart [plugins.security.ssl.http.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2026-04-22T09:45:30.785-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:30,785][WARN ][o.o.s.c.Salt ] [INDOT-DN01] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
2026-04-22T09:45:30.870-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:30,870][ERROR][o.o.s.a.s.SinkProvider ] [INDOT-DN01] Default endpoint could not be created, auditlog will not work properly.
2026-04-22T09:45:30.877-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:30,874][WARN ][o.o.s.a.r.AuditMessageRouter] [INDOT-DN01] No default storage available, audit log may not work properly. Please check configuration.
2026-04-22T09:45:30.878-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:30,874][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Message routing enabled: false
2026-04-22T09:45:30.960-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:30,958][INFO ][o.o.s.f.SecurityFilter ] [INDOT-DN01] indices are made immutable.
2026-04-22T09:45:31.407-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:31,407][INFO ][o.o.m.b.MLCircuitBreakerService] [INDOT-DN01] Registered ML memory breaker.
2026-04-22T09:45:31.409-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:31,409][INFO ][o.o.m.b.MLCircuitBreakerService] [INDOT-DN01] Registered ML disk breaker.
2026-04-22T09:45:31.410-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:31,410][INFO ][o.o.m.b.MLCircuitBreakerService] [INDOT-DN01] Registered ML native memory breaker.
2026-04-22T09:45:31.569-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:31,569][INFO ][o.r.Reflections ] [INDOT-DN01] Reflections took 89 ms to scan 1 urls, producing 21 keys and 59 values
2026-04-22T09:45:31.797-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:31,797][INFO ][o.o.a.b.ADCircuitBreakerService] [INDOT-DN01] Registered memory breaker.
2026-04-22T09:45:32.151-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,148][WARN ][stderr ] [INDOT-DN01] SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
2026-04-22T09:45:32.154-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,149][WARN ][stderr ] [INDOT-DN01] SLF4J: Defaulting to no-operation (NOP) logger implementation
2026-04-22T09:45:32.156-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,149][WARN ][stderr ] [INDOT-DN01] SLF4J: See SLF4J Error Codes for further details.
2026-04-22T09:45:32.670-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,670][INFO ][o.o.t.NettyAllocator ] [INDOT-DN01] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
2026-04-22T09:45:32.676-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,676][INFO ][o.o.s.s.t.SSLConfig ] [INDOT-DN01] SSL dual mode is disabled
2026-04-22T09:45:32.852-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:32,852][INFO ][o.o.d.DiscoveryModule ] [INDOT-DN01] using discovery type [zen] and seed hosts providers [settings, file]
2026-04-22T09:45:33.500-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:33,498][WARN ][o.o.g.DanglingIndicesState] [INDOT-DN01] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2026-04-22T09:45:34.232-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,232][INFO ][o.o.n.Node ] [INDOT-DN01] initialized
2026-04-22T09:45:34.243-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,233][INFO ][o.o.n.Node ] [INDOT-DN01] starting …
2026-04-22T09:45:34.418-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,418][INFO ][o.o.t.TransportService ] [INDOT-DN01] publish_address {10.10.40.7:9300}, bound_addresses {10.10.40.7:9300}
2026-04-22T09:45:34.421-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,421][INFO ][o.o.t.TransportService ] [INDOT-DN01] Remote clusters initialized successfully.
2026-04-22T09:45:34.664-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,664][INFO ][o.o.b.BootstrapChecks ] [INDOT-DN01] bound or publishing to a non-loopback address, enforcing bootstrap checks
2026-04-22T09:45:34.691-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,689][INFO ][o.o.c.c.Coordinator ] [INDOT-DN01] cluster UUID [p9MQt7xARgydDhNAk9_2LQ]
2026-04-22T09:45:34.938-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:34,936][INFO ][o.o.c.s.MasterService ] [INDOT-DN01] elected-as-cluster-manager ([1] nodes joined)[{INDOT-DN01}{UCR7rRE6QbCLt1fgj6DQgg}{hFQdgJv2RW6FDc7xZq8hTg}{10.10.40.7}{10.10.40.7:9300}{dimrs}{shard_indexing_pressure_enabled=true} elect leader, BECOME_CLUSTER_MANAGER_TASK, FINISH_ELECTION], term: 5, version: 51, delta: cluster-manager node changed {previous , current [{INDOT-DN01}{UCR7rRE6QbCLt1fgj6DQgg}{hFQdgJv2RW6FDc7xZq8hTg}{10.10.40.7}{10.10.40.7:9300}{dimrs}{shard_indexing_pressure_enabled=true}]}
2026-04-22T09:45:35.048-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,048][INFO ][o.o.c.s.ClusterApplierService] [INDOT-DN01] cluster-manager node changed {previous , current [{INDOT-DN01}{UCR7rRE6QbCLt1fgj6DQgg}{hFQdgJv2RW6FDc7xZq8hTg}{10.10.40.7}{10.10.40.7:9300}{dimrs}{shard_indexing_pressure_enabled=true}]}, term: 5, version: 51, reason: Publication{term=5, version=51}
2026-04-22T09:45:35.061-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,060][INFO ][o.o.i.i.ManagedIndexCoordinator] [INDOT-DN01] Cache cluster manager node onClusterManager time: 1776865535060
2026-04-22T09:45:35.071-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,071][INFO ][o.o.a.c.ADClusterEventListener] [INDOT-DN01] Cluster is not recovered yet.
2026-04-22T09:45:35.098-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,095][INFO ][o.o.d.PeerFinder ] [INDOT-DN01] setting findPeersInterval to [1s] as node commission status = [true] for local node [{INDOT-DN01}{UCR7rRE6QbCLt1fgj6DQgg}{hFQdgJv2RW6FDc7xZq8hTg}{10.10.40.7}{10.10.40.7:9300}{dimrs}{shard_indexing_pressure_enabled=true}]
2026-04-22T09:45:35.216-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,214][INFO ][o.o.h.AbstractHttpServerTransport] [INDOT-DN01] publish_address {10.10.40.7:9200}, bound_addresses {10.10.40.7:9200}
2026-04-22T09:45:35.221-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,219][INFO ][o.o.n.Node ] [INDOT-DN01] started
2026-04-22T09:45:35.222-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,220][INFO ][o.o.s.OpenSearchSecurityPlugin] [INDOT-DN01] Node started
2026-04-22T09:45:35.222-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,220][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Will attempt to create index .opendistro_security and default configs if they are absent
2026-04-22T09:45:35.234-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,231][INFO ][o.o.s.OpenSearchSecurityPlugin] [INDOT-DN01] 0 OpenSearch Security modules loaded so far:
2026-04-22T09:45:35.238-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,231][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Background init thread started. Install default config?: true
2026-04-22T09:45:35.248-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,247][INFO ][o.o.c.s.ClusterSettings ] [INDOT-DN01] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
2026-04-22T09:45:35.285-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,285][INFO ][o.o.a.c.HashRing ] [INDOT-DN01] Node added: [UCR7rRE6QbCLt1fgj6DQgg]
2026-04-22T09:45:35.291-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,289][INFO ][o.o.g.GatewayService ] [INDOT-DN01] recovered [6] indices into cluster_state
2026-04-22T09:45:35.299-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,295][INFO ][o.o.a.c.HashRing ] [INDOT-DN01] Add data node to AD version hash ring: UCR7rRE6QbCLt1fgj6DQgg
2026-04-22T09:45:35.307-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,304][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Index .opendistro_security already exists
2026-04-22T09:45:35.314-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,304][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Node started, try to initialize it. Wait for at least yellow cluster state…
2026-04-22T09:45:35.314-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,306][WARN ][o.o.o.i.ObservabilityIndex] [INDOT-DN01] message: index [.opensearch-observability/IPMlSzlCRB2Znoa1oZOJjA] already exists
2026-04-22T09:45:35.317-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,310][INFO ][o.o.a.c.HashRing ] [INDOT-DN01] All nodes with known AD version: {UCR7rRE6QbCLt1fgj6DQgg=ADNodeInfo{version=2.12.0, isEligibleDataNode=true}}
2026-04-22T09:45:35.319-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,312][INFO ][o.o.a.c.HashRing ] [INDOT-DN01] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
2026-04-22T09:45:35.322-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,312][INFO ][o.o.a.c.HashRing ] [INDOT-DN01] Build AD version hash ring successfully
2026-04-22T09:45:35.323-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,322][INFO ][o.o.a.c.ADDataMigrator ] [INDOT-DN01] Start migrating AD data
2026-04-22T09:45:35.328-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,323][INFO ][o.o.a.c.ADDataMigrator ] [INDOT-DN01] AD job index doesn’t exist, no need to migrate
2026-04-22T09:45:35.330-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,323][INFO ][o.o.a.c.ADClusterEventListener] [INDOT-DN01] Init AD version hash ring successfully
2026-04-22T09:45:35.746-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:35,746][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[graylog_0/6gUrtL17TXGPMRu2kDPB0A]
2026-04-22T09:45:36.286-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:36,283][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[gl-events_0/QijfprlnR9Skww3qEm_4aA]
2026-04-22T09:45:36.309-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:36,308][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[gl-system-events_1/rV4QJSciRdmDjXdw75-jDA]
2026-04-22T09:45:36.396-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:36,395][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[.opendistro_security/1dVTT6EHS6ufrDk3veQeXg]
2026-04-22T09:45:36.619-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:45:36.910-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:36,910][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘config’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.009-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,007][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[.plugins-ml-config/WFP8hw7qRQO7VISs7kxePg]
2026-04-22T09:45:37.055-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,054][INFO ][o.o.p.PluginsService ] [INDOT-DN01] PluginService:onIndexModule index:[.opensearch-observability/IPMlSzlCRB2Znoa1oZOJjA]
2026-04-22T09:45:37.128-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,127][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id config, skipping update.
2026-04-22T09:45:37.129-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,127][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘roles’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.153-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,151][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id roles, skipping update.
2026-04-22T09:45:37.155-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,151][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘rolesmapping’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.172-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,169][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id rolesmapping, skipping update.
2026-04-22T09:45:37.173-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,170][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘internalusers’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.209-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,209][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id internalusers, skipping update.
2026-04-22T09:45:37.210-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,209][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘actiongroups’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.224-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,222][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id actiongroups, skipping update.
2026-04-22T09:45:37.225-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,223][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘tenants’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.239-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,238][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id tenants, skipping update.
2026-04-22T09:45:37.240-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,238][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘nodesdn’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
2026-04-22T09:45:37.253-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,248][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id nodesdn, skipping update.
2026-04-22T09:45:37.256-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,248][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘whitelist’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
2026-04-22T09:45:37.257-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,253][INFO ][o.o.c.r.a.AllocationService] [INDOT-DN01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.opensearch-observability][0]]]).
2026-04-22T09:45:37.272-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,272][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id whitelist, skipping update.
2026-04-22T09:45:37.279-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,278][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘allowlist’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
2026-04-22T09:45:37.287-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,286][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id allowlist, skipping update.

7

2026-04-22T09:45:37.288-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,287][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Will update ‘audit’ with /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
2026-04-22T09:45:37.355-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,354][INFO ][o.o.s.s.ConfigHelper ] [INDOT-DN01] Index .opendistro_security already contains doc with id audit, skipping update.
2026-04-22T09:45:37.702-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,702][INFO ][stdout ] [INDOT-DN01] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
2026-04-22T09:45:37.705-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,703][INFO ][stdout ] [INDOT-DN01] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
2026-04-22T09:45:37.705-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,703][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing on REST API is enabled.
2026-04-22T09:45:37.705-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,704][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
2026-04-22T09:45:37.706-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,704][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing on Transport API is enabled.
2026-04-22T09:45:37.708-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,704][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
2026-04-22T09:45:37.709-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,704][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing of request body is enabled.
2026-04-22T09:45:37.712-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,710][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Bulk requests resolution is disabled during request auditing.
2026-04-22T09:45:37.712-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,710][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Index resolution is enabled during request auditing.
2026-04-22T09:45:37.712-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,710][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Sensitive headers auditing is enabled.
2026-04-22T09:45:37.712-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,711][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing requests from kibanaserver users is disabled.
2026-04-22T09:45:37.712-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,711][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing request headers is disabled.
2026-04-22T09:45:37.718-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,711][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing of external configuration is disabled.
2026-04-22T09:45:37.719-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,711][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing of internal configuration is enabled.
2026-04-22T09:45:37.720-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,711][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing only metadata information for read request is enabled.
2026-04-22T09:45:37.721-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,721][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing will watch {} for read requests.
2026-04-22T09:45:37.721-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,721][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing read operation requests from kibanaserver users is disabled.
2026-04-22T09:45:37.724-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,723][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing only metadata information for write request is enabled.
2026-04-22T09:45:37.724-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,723][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing diffs for write requests is disabled.
2026-04-22T09:45:37.724-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,723][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing write operation requests from kibanaserver users is disabled.
2026-04-22T09:45:37.724-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,723][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Auditing will watch for write requests.
2026-04-22T09:45:37.725-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,723][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] .opendistro_security is used as internal security index.
2026-04-22T09:45:37.726-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,725][INFO ][o.o.s.a.i.AuditLogImpl ] [INDOT-DN01] Internal index used for posting audit logs is null
2026-04-22T09:45:37.727-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,726][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Hot-reloading of audit configuration is enabled
2026-04-22T09:45:37.727-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:37,727][INFO ][o.o.s.c.ConfigurationRepository] [INDOT-DN01] Node ‘INDOT-DN01’ initialized
2026-04-22T09:45:45.100-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:45,100][INFO ][o.o.m.a.MLModelAutoReDeployer] [INDOT-DN01] Index not found, not performing auto reloading!
2026-04-22T09:45:45.102-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:45,102][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [INDOT-DN01] Starting ML sync up job…
2026-04-22T09:45:46.614-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:45:55.120-04:00 INFO [OpensearchProcessImpl] [2026-04-22T09:45:55,119][INFO ][o.o.m.c.MLSyncUpCron ] [INDOT-DN01] ML configuration already initialized, no action needed
2026-04-22T09:45:56.614-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:46:06.614-04:00 INFO [ClusterNodeStateTracer] Updating cluster node 65e90c32-e9b6-4128-afe0-5e67dbf36fc2 from STARTING to UNAVAILABLE (reason: HEALTH_CHECK_FAILED)
2026-04-22T09:46:06.623-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:46:16.614-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
2026-04-22T09:46:26.614-04:00 WARN [OpensearchNodeHeartbeat] Opensearch REST api of process 38424 unavailable. Cause: Connection refused
^C

8

Hey @chilla67,

Has anything changed from a networking perspective that has not been accounted for in the Datanode.conf during the migration? Do the Datanode certs need to be renewed?

Could you post your redacted datanode.conf.

9

Hello,

This did have to get moved to a new system with a new network infrastructure, however the network/subnet im using stayed the same in my lab and when it moved to production. When I migrated, I did have to re enter the IPs for the VMs after they were converted to hyper-v but I ensured that each VM got the same IP it had when it was working in the lab. Possibly they could need new Certs? They were just deployed with Certs right before I migrated, so I dont suspect them to be expired, though im not an expert at graylog so I cant rule out certs arent the issue.

#####################################

GRAYLOG DATANODE CONFIGURATION FILE

#####################################

This is the Graylog DataNode configuration file. The file has to use ISO 8859-1/Latin-1 character encoding.

Characters that cannot be directly represented in this encoding can be written using Unicode escapes

as defined in Chapter 3. Lexical Structure , using the \u prefix.

For example, \u002c.

* Entries are generally expected to be a single line of the form, one of the following:

propertyName=propertyValue

propertyName:propertyValue

* White space that appears between the property name and property value is ignored,

so the following are equivalent:

name=Stephen

name = Stephen

* White space at the beginning of the line is also ignored.

* Lines that start with the comment characters ! or # are ignored. Blank lines are also ignored.

* The property value is generally terminated by the end of the line. White space following the

property value is not ignored, and is treated as part of the property value.

* A property value can span several lines if each line is terminated by a backslash (‘\’) character.

For example:

targetCities=\

Detroit,\

Chicago,\

Los Angeles

This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).

* The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.

* The backslash character must be escaped as a double backslash. For example:

path=c:\docs\doc1

The auto-generated node ID will be stored in this file and read after restarts. It is a good idea

to use an absolute file path here if you are starting Graylog DataNode from init scripts or similar.

node_id_file = /etc/graylog/datanode/node-id

location of your data-node configuration files - put additional files like manually created certificates etc. here

config_location = /etc/graylog/datanode

You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.

Generate one by using for example: pwgen -N 1 -s 96

ATTENTION: This value must be the same on all Graylog and Datanode nodes in the cluster.

Changing this value after installation will render all user sessions and encrypted values in the database invalid. (e.g. encrypted access tokens)

password_secret = REDACTED

The default root user is named ‘admin’

#root_username = admin

You MUST specify a hash password for the root user (which you only need to initially set up the

system and in case you lose connectivity to your authentication backend)

This password cannot be changed using the API or via the web interface. If you need to change it,

modify it in this file.

Create one by using for example: echo -n yourpassword | shasum -a 256

and put the resulting hash value into the following line

root_password_sha2 = REDACTED

connection to MongoDB, shared with the Graylog server

See Connection Strings - Database Manual - MongoDB Docs for details

mongodb_uri = mongodb://INDOT-MONGO01:27017,INDOT-MONGO02:27017,INDOT-MONGO03:27017/graylog?replicaSet=rs0

HTTP bind address

The network interface used by the Graylog DataNode to bind all services.

bind_address = 10.10.40.7
graylog_server = http://INDOT-GL01:9000

Hostname

HTTP port

The port where the DataNode REST api is listening

datanode_http_port = 8999

HTTP publish URI

This configuration should be used if you want to connect to this Graylog DataNode’s REST API and it is available on

another network interface than $http_bind_address,

for example if the machine has multiple network interfaces or is behind a NAT gateway.

http_publish_uri =

OpenSearch HTTP port

The port where OpenSearch HTTP is listening on

opensearch_http_port = 9200

OpenSearch transport port

The port where OpenSearch transports is listening on

opensearch_transport_port = 9300

OpenSearch node name config option

use this, if your node name should be different from the hostname that’s found by programmatically looking it up

node_name = INDOT-DN01

OpenSearch discovery_seed_hosts config option

if you’re not using the automatic data node setup and want to create a cluster, you have to setup the discovery seed hosts

opensearch_discovery_seed_hosts = 10.10.40.7,10.10.40.8,10.10.40.9

OpenSearch initial_manager_nodes config option

if you’re not using the automatic data node setup and want to create a cluster, you have to setup the initial manager nodes

make sure to remove this setting after the cluster has formed

initial_cluster_manager_nodes = INDOT-DN01, INDOT-DN02, INDOT-DN03

OpenSearch folders

set these if you need OpenSearch to be located in a special place or want to include an existing version

Root directory of the used opensearch distribution

opensearch_location = /usr/share/graylog-datanode/dist

opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch

OpenSearch Certificate bundles for transport and http layer security

if you’re not using the automatic data node setup, you can manually configure your SSL certificates

transport_certificate = datanode-transport-certificates.p12

transport_certificate_password = password

http_certificate = datanode-http-certificates.p12

http_certificate_password = password

OpenSearch log buffers size

the number of lines from stderr and stdout of the OpenSearch process that are buffered inside the DataNode for logging etc.

process_logs_buffer_size = 500

OpenSearch JWT token usage

communication between Graylog and OpenSearch is secured by JWT. These are the defaults used for the token usage

adjust them, if you have special needs.

indexer_jwt_auth_token_caching_duration = 60s

indexer_jwt_auth_token_expiration_duration = 180s

10

@Wine_Merchant Forgot to mention you. Here is my conf for the data nodes. ^

11

Under System/Cluster Config are you able to regenerate the certs? This might not be the issue as there are no errors indicating a cert problem.

Could you try binding to 0.0.0.0 instead of the explicit IP?

How is network connectivity between graylog to data node and data node to data node?

12

You are a life saver!! setting the bind address to 0.0.0.0 now shows all 3 of my datanodes available in the UI. I spent the past week trying to figure it out lol.

Thank you so much for the help!

13

Perhaps something amiss with the network adapter post migration, either way glad it’s working.

15

Hey, I ran into something very similar when moving a Graylog setup between hypervisors, and in my case the issue ended up being related to how the datanodes re-register and communicate after the migration.

A couple of things you might want to double-check:

  • Node identity / certificates: When you moved from VMware to Hyper-V, some underlying identifiers (hostname resolution, IP bindings, or TLS cert fingerprints) may have changed. Datanodes can show as “Unavailable” if they can’t properly authenticate or match what Graylog expects.

  • Network + bind addresses: Even if everything is on the same subnet, verify that each datanode is still binding to the correct interface/IP and that Graylog is pointing to the updated addresses (no leftover VMware-era configs).

  • Cluster state in MongoDB: Since you already tried resetting nodes, make sure there are no stale entries left for the old node IDs. Sometimes it’s cleaner to fully remove and re-add datanodes rather than partially reconfiguring.

  • Time sync: Sounds basic, but after migrations I’ve seen NTP drift cause weird “unavailable” states due to auth/token issues.

  • Firewall / ports: Hyper-V environments sometimes default to stricter rules. Double-check required ports between Graylog and datanodes are खुल (especially internal cluster comms).

If you haven’t already, try bringing up just one datanode fresh (completely wiped config + re-registered) and confirm it becomes healthy before scaling back to three.

Also, while troubleshooting cluster setups like this, I’ve found it helpful to quickly spin up consistent naming/host patterns so configs stay clean—tools like generate name can actually help avoid duplication/confusion when rebuilding nodes.

Let us know what your datanode logs say after restart—there’s usually a pretty specific clue there