1. Describe your incident:
GrayLog 6.1 datanode would not start.
2. Describe your environment:
- OS Information:
Debian 12.9
- Package Version:
What is it?
- Service logs, configurations, and environment variables:
2025-02-13T18:45:31.066+02:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-02-13T18:45:31.461+02:00 INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-02-13T18:45:31.645+02:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.1.0-31-amd64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@615e3f51, com.mongodb.Jep395RecordCodecProvider@407873d3, com.mongodb.KotlinCodecProvider@608bc8f8]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-13T18:45:31.649+02:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.1.0-31-amd64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@615e3f51, com.mongodb.Jep395RecordCodecProvider@407873d3, com.mongodb.KotlinCodecProvider@608bc8f8]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-13T18:45:31.665+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18466918, minRoundTripTimeNanos=0}
2025-02-13T18:45:31.732+02:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 7.0.16
2025-02-13T18:45:31.942+02:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.1.0-31-amd64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@615e3f51, com.mongodb.Jep395RecordCodecProvider@407873d3, com.mongodb.KotlinCodecProvider@608bc8f8]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-13T18:45:31.943+02:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.1.0-31-amd64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@615e3f51, com.mongodb.Jep395RecordCodecProvider@407873d3, com.mongodb.KotlinCodecProvider@608bc8f8]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-13T18:45:31.944+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1331318, minRoundTripTimeNanos=0}
2025-02-13T18:45:31.949+02:00 INFO [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29997 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-02-13T18:45:31.985+02:00 INFO [DatanodeDirectories] Opensearch of the node e9eb9b9a-d087-4062-abf4-81136e22435c uses following directories as its storage: DatanodeDirectories{dataTargetDir=‘/var/lib/graylog-datanode/opensearch/data’, logsTargetDir=‘/var/log/graylog-datanode/opensearch’, configurationSourceDir=‘Optional[/etc/graylog/datanode]’, configurationTargetDir=‘/var/lib/graylog-datanode/opensearch/config’, opensearchProcessConfigurationDir=‘/var/lib/graylog-datanode/opensearch/config/opensearch’}
2025-02-13T18:45:32.018+02:00 INFO [OpensearchConfigSync] Directory used for Opensearch process configuration is /var/lib/graylog-datanode/opensearch/config/opensearch
2025-02-13T18:45:32.037+02:00 INFO [OpensearchConfigSync] Synchronizing Opensearch configuration
2025-02-13T18:45:32.041+02:00 INFO [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-02-13T18:45:32.055+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-02-13T18:45:32.056+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-02-13T18:45:32.056+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-02-13T18:45:32.056+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-02-13T18:45:32.057+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-02-13T18:45:32.057+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-02-13T18:45:32.057+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-02-13T18:45:32.057+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-02-13T18:45:32.058+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-02-13T18:45:32.058+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-02-13T18:45:32.058+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-02-13T18:45:32.059+02:00 INFO [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-02-13T18:45:32.059+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-02-13T18:45:32.060+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12
2025-02-13T18:45:32.060+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/http-keystore.p12
2025-02-13T18:45:32.060+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-02-13T18:45:32.061+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12
2025-02-13T18:45:32.061+02:00 INFO [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-02-13T18:45:32.064+02:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch
2025-02-13T18:45:32.065+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-02-13T18:45:32.066+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-02-13T18:45:32.066+02:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-02-13T18:45:32.067+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-02-13T18:45:32.068+02:00 INFO [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-02-13T18:45:32.068+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-02-13T18:45:32.069+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-02-13T18:45:32.069+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-02-13T18:45:32.070+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-02-13T18:45:32.071+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-02-13T18:45:32.071+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-02-13T18:45:32.072+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-02-13T18:45:32.073+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-02-13T18:45:32.073+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-02-13T18:45:32.074+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-02-13T18:45:32.074+02:00 INFO [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-02-13T18:45:32.077+02:00 INFO [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-02-13T18:45:32.091+02:00 INFO [OpensearchDataDirCompatibilityCheck] Found 0 indices and all of them are valid with current opensearch version 2.15.0
2025-02-13T18:45:32.364+02:00 INFO [DatanodeDirectories] Opensearch of the node e9eb9b9a-d087-4062-abf4-81136e22435c uses following directories as its storage: DatanodeDirectories{dataTargetDir=‘/var/lib/graylog-datanode/opensearch/data’, logsTargetDir=‘/var/log/graylog-datanode/opensearch’, configurationSourceDir=‘Optional[/etc/graylog/datanode]’, configurationTargetDir=‘/var/lib/graylog-datanode/opensearch/config’, opensearchProcessConfigurationDir=‘/var/lib/graylog-datanode/opensearch/config/opensearch’}
2025-02-13T18:45:34.102+02:00 INFO [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 1.554 s
2025-02-13T18:45:34.148+02:00 INFO [ServerBootstrap] Graylog datanode 6.1.6+a644883 starting up
2025-02-13T18:45:34.148+02:00 INFO [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-31-amd64
2025-02-13T18:45:34.149+02:00 INFO [ServerBootstrap] Deployment: deb
2025-02-13T18:45:34.149+02:00 INFO [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-13T18:45:34.149+02:00 INFO [ServerBootstrap] Arch: amd64
2025-02-13T18:45:34.182+02:00 INFO [PeriodicalsService] Starting 6 periodicals …
2025-02-13T18:45:34.183+02:00 INFO [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader …
2025-02-13T18:45:34.184+02:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-13T18:45:34.193+02:00 INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-13T18:45:34.193+02:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-13T18:45:34.200+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-13T18:45:34.204+02:00 INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-13T18:45:34.437+02:00 INFO [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-13T18:45:34.567+02:00 INFO [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, 0:0:0:0:0:0:0:1, ${HOSTNAME}
2025-02-13T18:45:34.568+02:00 INFO [OpensearchSecurityConfiguration] Opensearch transport certificate has following serial number: 1739444348239
2025-02-13T18:45:34.569+02:00 INFO [OpensearchSecurityConfiguration] Opensearch transport certificate has following validity: 2025-02-13T12:59:08.000+0200 - 2035-02-11T12:59:08.000+0200
2025-02-13T18:45:34.589+02:00 INFO [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: 127.0.0.1, 0:0:0:0:0:0:0:1, localhost, 0:0:0:0:0:0:0:1, ${HOSTNAME}
2025-02-13T18:45:34.590+02:00 INFO [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following serial number: 1739444348239
2025-02-13T18:45:34.590+02:00 INFO [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following validity: 2025-02-13T12:59:08.000+0200 - 2035-02-11T12:59:08.000+0200
2025-02-13T18:45:34.596+02:00 INFO [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-02-13T18:45:34.636+02:00 INFO [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
2025-02-13T18:45:34.636+02:00 INFO [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2025-02-13T18:45:34.637+02:00 INFO [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical].
2025-02-13T18:45:34.638+02:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[guava-33.3.1-jre.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[guava-33.3.1-jre.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[guava-33.3.1-jre.jar:?]
at org.graylog.datanode.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:228) [graylog-datanode.jar:?]
at org.graylog.datanode.bootstrap.CmdLineTool.doRun(CmdLineTool.java:305) [graylog-datanode.jar:?]
at org.graylog.datanode.bootstrap.CmdLineTool.run(CmdLineTool.java:246) [graylog-datanode.jar:?]
at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
Caused by: org.graylog.datanode.configuration.OpensearchConfigurationException: java.security.KeyStoreException: Keystore does not contain root X509Certificate in the certificate chain!
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:183) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:218) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:95) ~[graylog-datanode.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[guava-33.3.1-jre.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.3.1-jre.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.security.KeyStoreException: Keystore does not contain root X509Certificate in the certificate chain!
at org.graylog.datanode.configuration.TruststoreCreator.lambda$findRootCert$4(TruststoreCreator.java:106) ~[graylog-datanode.jar:?]
at java.base/java.util.Optional.orElseThrow(Unknown Source) ~[?:?]
at org.graylog.datanode.configuration.TruststoreCreator.findRootCert(TruststoreCreator.java:106) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.TruststoreCreator.addRootCert(TruststoreCreator.java:65) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.variants.OpensearchSecurityConfiguration.configure(OpensearchSecurityConfiguration.java:92) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:159) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:218) ~[graylog-datanode.jar:?]
at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:95) ~[graylog-datanode.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[guava-33.3.1-jre.jar:?]
at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.3.1-jre.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-13T18:45:34.646+02:00 INFO [Server] SIGNAL received. Shutting down.
2025-02-13T18:45:34.647+02:00 INFO [GracefulShutdown] Graceful shutdown initiated.
2025-02-13T18:45:34.647+02:00 INFO [GracefulShutdown] Goodbye.
3. What steps have you already taken to try and solve the problem?
Absolutely everything I could find on the internet.
4. How can the community help?
How to make datanode run?
