Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident: I am attempting to do an in-place migration from OpenSearch 2.19.2-1 to Graylog Data Node 7.0.5-2 using the documentation available at Data Node Migration
2. Describe your environment:
-
OS Information: Rocky Linux 8.10 (Green Obsidian)
-
Package Version: OpenSearch 2.19.2-1 (currently running), Graylog Data Node 7.0.5-2
-
Service logs, configurations, and environment variables:
Graylog datanode conf:
node_id_file = /var/lib/graylog-datanode/node-id
node_name = my-opensrch-01
password_secret = <actual, correct graylog secret>
mongodb_uri = mongodb://graylog:graylog-nice-pw@my-graylog-01.mydomain.com:27017,my-graylog-02.mydomain.com:27017,my-graylog-03.mydomain.com:27017/graylog?replicaSet=rs0
bind_address = 10.10.0.237
datanode_http_port = 8999
http_publish_uri = http://10.10.0.237:8999/
clustername = graylog-search
node_roles = data,cluster_manager
initial_cluster_manager_nodes = my-opensrch-01.mydomain.com,my-opensrch-02.mydomain.com,my-opensrch-03.mydomain.com
opensearch_discovery_seed_hosts = 10.10.0.237,10.10.0.238,10.10.0.239
opensearch_network_host = 10.10.0.237
opensearch_http_port = 9200
opensearch_transport_port = 9300
opensearch_location = /usr/share/opensearch/
opensearch_data_location = /mnt/opensearch/
opensearch_logs_location = /var/log/graylog-datanode/opensearch/
Datanode.log file:
2026-03-30T09:28:24.312-07:00 WARN [CmdLineTool] Couldn't create native lib dir </datanode/config/native_libs>. Unable to set ZstdTempFolder system property.
java.nio.file.AccessDeniedException: /datanode
at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source)
at java.base/java.nio.file.Files.createDirectory(Unknown Source)
at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source)
at java.base/java.nio.file.Files.createDirectories(Unknown Source)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:310)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
2026-03-30T09:28:24.408-07:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2026-03-30T09:28:24.410-07:00 WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2026-03-30T09:28:24.649-07:00 INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2026-03-30T09:28:24.878-07:00 INFO [cluster] Adding discovered server my-graylog-01.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:24.890-07:00 INFO [cluster] Adding discovered server my-graylog-02.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:24.890-07:00 INFO [cluster] Adding discovered server my-graylog-03.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:24.906-07:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2262d6d5, com.mongodb.Jep395RecordCodecProvider@40de8f93, com.mongodb.KotlinCodecProvider@6ff0b1cc, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[my-graylog-01.mydomain.com:27017, my-graylog-02.mydomain.com:27017, my-graylog-03.mydomain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2026-03-30T09:28:24.907-07:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2262d6d5, com.mongodb.Jep395RecordCodecProvider@40de8f93, com.mongodb.KotlinCodecProvider@6ff0b1cc, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[my-graylog-01.mydomain.com:27017, my-graylog-02.mydomain.com:27017, my-graylog-03.mydomain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2026-03-30T09:28:24.928-07:00 INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 5. Remaining time: 29995 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=my-graylog-03.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=my-graylog-01.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=my-graylog-02.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}].
2026-03-30T09:28:24.938-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-02.mydomain.com:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=15526300, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-02:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269263784977}
2026-03-30T09:28:24.938-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-01.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=16226075, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-01.mydomain.com:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269263765349}
2026-03-30T09:28:24.938-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-03.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=15480240, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-03:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269263762203}
2026-03-30T09:28:24.940-07:00 INFO [cluster] Adding discovered server my-graylog-03:27017 to client view of cluster
2026-03-30T09:28:24.941-07:00 INFO [cluster] Adding discovered server my-graylog-02:27017 to client view of cluster
2026-03-30T09:28:24.941-07:00 INFO [cluster] Server my-graylog-03.mydomain.com:27017 is no longer a member of the replica set. Removing from client view of cluster.
2026-03-30T09:28:24.943-07:00 INFO [cluster] Server my-graylog-02.mydomain.com:27017 is no longer a member of the replica set. Removing from client view of cluster.
2026-03-30T09:28:24.943-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-03:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=948417, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-03:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269280325280}
2026-03-30T09:28:24.943-07:00 INFO [cluster] Discovered replica set primary my-graylog-02.mydomain.com:27017 with max election id 7fffffff00000000000000a2 and max set version 5
2026-03-30T09:28:24.944-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-02:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=924691, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-02:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269281636209}
2026-03-30T09:28:24.945-07:00 INFO [cluster] Discovered replica set primary my-graylog-02:27017 with max election id 7fffffff00000000000000a2 and max set version 5
2026-03-30T09:28:25.001-07:00 INFO [DatanodeDirectories] Opensearch of the node 2e68dfab-c04b-40d6-a7c4-451dc771202a uses following directories as its storage: DatanodeDirectories{dataTargetDir='/mnt/opensearch', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional.empty', configurationTargetDir='/datanode/config'}
2026-03-30T09:28:25.028-07:00 INFO [cluster] Adding discovered server my-graylog-01.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:25.028-07:00 INFO [cluster] Adding discovered server my-graylog-02.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:25.029-07:00 INFO [cluster] Adding discovered server my-graylog-03.mydomain.com:27017 to client view of cluster
2026-03-30T09:28:25.029-07:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2262d6d5, com.mongodb.Jep395RecordCodecProvider@40de8f93, com.mongodb.KotlinCodecProvider@6ff0b1cc, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[my-graylog-01.mydomain.com:27017, my-graylog-02.mydomain.com:27017, my-graylog-03.mydomain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2026-03-30T09:28:25.030-07:00 INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/21.0.10+7-LTS"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylog', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2262d6d5, com.mongodb.Jep395RecordCodecProvider@40de8f93, com.mongodb.KotlinCodecProvider@6ff0b1cc, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[my-graylog-01.mydomain.com:27017, my-graylog-02.mydomain.com:27017, my-graylog-03.mydomain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs0', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2026-03-30T09:28:25.030-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-01.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=904852, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-01.mydomain.com:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5729ebb9e82ecc9da539c, counter=10}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269367546992}
2026-03-30T09:28:25.030-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-02.mydomain.com:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=775963, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-02:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269367749064}
2026-03-30T09:28:25.031-07:00 INFO [cluster] Adding discovered server my-graylog-03:27017 to client view of cluster
2026-03-30T09:28:25.031-07:00 INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 24. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=my-graylog-03.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=my-graylog-01.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=my-graylog-02.mydomain.com:27017, type=UNKNOWN, state=CONNECTING}].
2026-03-30T09:28:25.031-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-03.mydomain.com:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=827242, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-03:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269368087439}
2026-03-30T09:28:25.031-07:00 INFO [cluster] Adding discovered server my-graylog-02:27017 to client view of cluster
2026-03-30T09:28:25.032-07:00 INFO [cluster] Server my-graylog-03.mydomain.com:27017 is no longer a member of the replica set. Removing from client view of cluster.
2026-03-30T09:28:25.032-07:00 INFO [cluster] Server my-graylog-02.mydomain.com:27017 is no longer a member of the replica set. Removing from client view of cluster.
2026-03-30T09:28:25.032-07:00 INFO [cluster] Discovered replica set primary my-graylog-02.mydomain.com:27017 with max election id 7fffffff00000000000000a2 and max set version 5
2026-03-30T09:28:25.033-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-02:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1181238, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-02:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000000a2, setVersion=5, topologyVersion=TopologyVersion{processId=69c5734b7911307697c7fb33, counter=7}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269370643219}
2026-03-30T09:28:25.033-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=my-graylog-03:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=27, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1308033, minRoundTripTimeNanos=0, setName='rs0', canonicalAddress=my-graylog-03:27017, hosts=[my-graylog-03:27017, my-graylog-01.mydomain.com:27017, my-graylog-02:27017], passives=[], arbiters=[], primary='my-graylog-02:27017', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=69c5736d190a4b6ade386cca, counter=5}, lastWriteDate=Mon Mar 30 09:28:24 PDT 2026, lastUpdateTimeNanos=36535269370654100}
2026-03-30T09:28:25.034-07:00 INFO [cluster] Discovered replica set primary my-graylog-02:27017 with max election id 7fffffff00000000000000a2 and max set version 5
2026-03-30T09:28:25.060-07:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 8.2.5
2026-03-30T09:28:25.062-07:00 ERROR [CmdLineTool] Startup error:
java.lang.IllegalArgumentException: Could not detect any opensearch distribution. Directory used for Opensearch detection: /usr/share/opensearch. Please configure opensearch_location to a directory that contains an opensearch distribution for your architecture x64. You can download Opensearch from https://opensearch.org/downloads.html . Please extract the downloaded distribution and point opensearch_location configuration option to that directory.
at org.graylog.datanode.configuration.OpensearchDistributionProvider.createErrorMessage(OpensearchDistributionProvider.java:104)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.createErrorMessage(OpensearchDistributionProvider.java:98)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.detectInSubdirectory(OpensearchDistributionProvider.java:87)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.lambda$detectInDirectory$1(OpensearchDistributionProvider.java:69)
at java.base/java.util.Optional.orElseGet(Unknown Source)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.detectInDirectory(OpensearchDistributionProvider.java:69)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.lambda$new$0(OpensearchDistributionProvider.java:55)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:201)
at org.graylog.datanode.configuration.OpensearchDistributionProvider.get(OpensearchDistributionProvider.java:60)
at org.graylog.datanode.bootstrap.preflight.OpensearchBinPreflightCheck.runCheck(OpensearchBinPreflightCheck.java:52)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421)
at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:81)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.beforeInjectorCreation(DatanodeBootstrap.java:76)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:362)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
3. What steps have you already taken to try and solve the problem?
I have dug through the documentation extensively, I have tried various paths for the Opensearch_location setting, all based off of the output from rpm -ql opensearch, which returns 3 primary directories:
/etc/opensearch
/usr/share/opensearch
/var/lib/opensearch
Opensearch is configured as specified in the graylog 6.3 documentation: the data and log directories are chown’d opensearch:opensearch and have been for quite some time, this is still a currently working configuration
The contents of /mnt/opensearch:
ls -lah /mnt/opensearch/
total 20K
drwxr-xr-x. 3 opensearch opensearch 195 Jan 24 2025 .
drwxr-xr-x. 5 root root 62 Feb 20 18:29 ..
-rw-r--r--. 1 opensearch opensearch 5 May 30 2025 batch_metrics_enabled.conf
-rw-r--r--. 1 opensearch opensearch 5 May 30 2025 logging_enabled.conf
drwxr-xr-x. 3 opensearch opensearch 15 Jan 24 2025 nodes
-rw-r--r--. 1 opensearch opensearch 5 May 30 2025 performance_analyzer_enabled.conf
-rw-r--r--. 1 opensearch opensearch 5 May 30 2025 rca_enabled.conf
-rw-r--r--. 1 opensearch opensearch 5 May 30 2025 thread_contention_monitoring_enabled.conf
4. How can the community help?
What am I missing at this point? I have the data directory pointed to where the current opensearch data lives, I have the opensearch_location pointed at what I believe is the distribution location, I have a currently working opensearch cluster, serving a currently working Graylog 7.0.5 cluster. I can’t for the life of me figure out why it’s trying to create /datanode, nor do I know how to fix that. I need to have this use /mnt/opensearch - and if I have to move things around a bit, that’s fine- /mnt/opensearch is a dedicated storage device specifically for opensearch, and I need the data node to use it as well. if I need to create its own subdirectory off of that, I can do that- but I still have no idea how to configure the data node to use it. I also cannot get the data node to start.
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]