1. Describe your incident: I have been running Graylog 6.3 in docker fine for a while now, and most recently I upgraded to 7.0, including the data-node. Everything went just fine, from what I could tell, but all of a sudden, I’ve noticed that the data-node won’t start now.
2. Describe your environment:
-
OS Information: Ubuntu 24.04.3 // Docker version 29.0.1, build eedd969
-
Package Version: 7.0.0, I think.
-
Service logs, configurations, and environment variables:
2025-11-17T15:30:59.887Z INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-11-17T15:30:59.939Z WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-11-17T15:31:00.335Z INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-11-17T15:31:00.790Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-87-generic"}, "platform": "Java/Eclipse Adoptium/21.0.8+9-LTS", "env": {"container": {"runtime": "docker"}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2ffb3aec, com.mongodb.Jep395RecordCodecProvider@786ff1cb, com.mongodb.KotlinCodecProvider@46039a21, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-11-17T15:31:00.792Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-87-generic"}, "platform": "Java/Eclipse Adoptium/21.0.8+9-LTS", "env": {"container": {"runtime": "docker"}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2ffb3aec, com.mongodb.Jep395RecordCodecProvider@786ff1cb, com.mongodb.KotlinCodecProvider@46039a21, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-11-17T15:31:00.824Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=20973875, minRoundTripTimeNanos=0}
2025-11-17T15:31:00.826Z INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 3. Remaining time: 29993 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb:27017, type=UNKNOWN, state=CONNECTING}].
2025-11-17T15:31:00.890Z INFO [DatanodeDirectories] Opensearch of the node 78dafb63-9a67-417b-9cf9-89664eee18af uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/lib/graylog-datanode/opensearch/logs', configurationSourceDir='Optional.empty', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-11-17T15:31:00.932Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-87-generic"}, "platform": "Java/Eclipse Adoptium/21.0.8+9-LTS", "env": {"container": {"runtime": "docker"}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2ffb3aec, com.mongodb.Jep395RecordCodecProvider@786ff1cb, com.mongodb.KotlinCodecProvider@46039a21, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-11-17T15:31:00.933Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.6.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.0-87-generic"}, "platform": "Java/Eclipse Adoptium/21.0.8+9-LTS", "env": {"container": {"runtime": "docker"}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2ffb3aec, com.mongodb.Jep395RecordCodecProvider@786ff1cb, com.mongodb.KotlinCodecProvider@46039a21, EnumCodecProvider{}]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-11-17T15:31:00.934Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1143058, minRoundTripTimeNanos=0}
2025-11-17T15:31:00.934Z INFO [cluster] Waiting for server to become available for operation { ping: 1 } with ID 10. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb:27017, type=UNKNOWN, state=CONNECTING}].
2025-11-17T15:31:00.965Z INFO [MongoDBPreflightCheck] Connected to MongoDB version 7.0.25
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
2025-11-17T15:31:01.071Z INFO [MemorySegmentIndexInputProvider] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2025-11-17T15:31:01.145Z ERROR [CmdLineTool] Startup error:
org.graylog.datanode.filesystem.index.IndexerInformationParserException: No state file available in dir /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/iGKTdCeuTuSPEFclJq11fw
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.lambda$findStateFile$9(IndicesDirectoryParser.java:136)
at java.base/java.util.Optional.orElseThrow(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.findStateFile(IndicesDirectoryParser.java:136)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.getState(IndicesDirectoryParser.java:102)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parseIndex(IndicesDirectoryParser.java:108)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parseNode(IndicesDirectoryParser.java:94)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toList(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parse(IndicesDirectoryParser.java:77)
at org.graylog.datanode.bootstrap.preflight.OpensearchDataDirCompatibilityCheck.runCheck(OpensearchDataDirCompatibilityCheck.java:59)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421)
at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:81)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.beforeInjectorCreation(DatanodeBootstrap.java:76)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:362)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
Exception in thread "main" org.graylog.datanode.filesystem.index.IndexerInformationParserException: No state file available in dir /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/iGKTdCeuTuSPEFclJq11fw
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.lambda$findStateFile$9(IndicesDirectoryParser.java:136)
at java.base/java.util.Optional.orElseThrow(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.findStateFile(IndicesDirectoryParser.java:136)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.getState(IndicesDirectoryParser.java:102)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parseIndex(IndicesDirectoryParser.java:108)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parseNode(IndicesDirectoryParser.java:94)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toArray(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.toList(Unknown Source)
at org.graylog.datanode.filesystem.index.IndicesDirectoryParser.parse(IndicesDirectoryParser.java:77)
at org.graylog.datanode.bootstrap.preflight.OpensearchDataDirCompatibilityCheck.runCheck(OpensearchDataDirCompatibilityCheck.java:59)
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421)
at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:81)
at org.graylog.datanode.bootstrap.DatanodeBootstrap.beforeInjectorCreation(DatanodeBootstrap.java:76)
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:362)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:287)
at org.graylog.datanode.bootstrap.Main.main(Main.java:59)
3. What steps have you already taken to try and solve the problem? I’m not even sure where to begin. It was working just fine, and now it’s not. I don’t think I changed anything (although the Docker version did get upgraded)/
4. How can the community help? Does anyone have an idea what I could even start to do to troubleshoot with what’s going on?