Graylog fails to connect to Opensearch on port 9200

Graylog Central (peer support)
1

Similar to this one: Datanode not listening on 9200 - no opensearch

Except in my case I’m upgrading from Graylog 5.x to 6.x, and when I try to open the webbrowser, it simply fails to load.

Running on Debian 12, LXC container. Graylog 5.2 worked fine for a very long time.

The datanode log:

/var/log/graylog-datanode/datanode.log 
2025-04-04T18:46:08.864-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-04-04T18:46:09.194-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms2g -Xmx2g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-04-04T18:46:09.349-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@a565cbd, com.mongodb.Jep395RecordCodecProvider@6fc29daa, com.mongodb.KotlinCodecProvider@2f66f0d0]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.353-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@a565cbd, com.mongodb.Jep395RecordCodecProvider@6fc29daa, com.mongodb.KotlinCodecProvider@2f66f0d0]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.366-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=15772835, minRoundTripTimeNanos=0}
2025-04-04T18:46:09.425-05:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.18
2025-04-04T18:46:09.588-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@a565cbd, com.mongodb.Jep395RecordCodecProvider@6fc29daa, com.mongodb.KotlinCodecProvider@2f66f0d0]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.588-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1099932, minRoundTripTimeNanos=0}
2025-04-04T18:46:09.590-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@a565cbd, com.mongodb.Jep395RecordCodecProvider@6fc29daa, com.mongodb.KotlinCodecProvider@2f66f0d0]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.616-05:00 INFO  [DatanodeDirectories] Opensearch of the node 65745f4a-db8c-49a7-92ed-e4468e7e5b4a uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-04-04T18:46:09.642-05:00 INFO  [OpensearchConfigSync] Directory used for Opensearch process configuration is /var/lib/graylog-datanode/opensearch/config/opensearch
2025-04-04T18:46:09.656-05:00 INFO  [OpensearchConfigSync] Synchronizing Opensearch configuration
2025-04-04T18:46:09.658-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-04-04T18:46:09.658-05:00 INFO  [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-04-04T18:46:09.668-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-04-04T18:46:09.668-05:00 INFO  [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-04-04T18:46:09.670-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-04-04T18:46:09.670-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-04-04T18:46:09.671-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-04-04T18:46:09.671-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-04-04T18:46:09.671-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-04-04T18:46:09.671-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-04-04T18:46:09.671-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-04-04T18:46:09.672-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-04-04T18:46:09.672-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-04-04T18:46:09.672-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-04-04T18:46:09.672-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-04-04T18:46:09.673-05:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-04-04T18:46:09.675-05:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch
2025-04-04T18:46:09.676-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-04-04T18:46:09.677-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-04-04T18:46:09.677-05:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-04-04T18:46:09.678-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-04-04T18:46:09.679-05:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-04-04T18:46:09.679-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-04-04T18:46:09.680-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-04-04T18:46:09.680-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-04-04T18:46:09.681-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-04-04T18:46:09.681-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-04-04T18:46:09.682-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-04-04T18:46:09.683-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-04-04T18:46:09.683-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-04-04T18:46:09.684-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-04-04T18:46:09.685-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-04-04T18:46:09.685-05:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-04-04T18:46:09.687-05:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-04-04T18:46:09.906-05:00 INFO  [DatanodeDirectories] Opensearch of the node 65745f4a-db8c-49a7-92ed-e4468e7e5b4a uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-04-04T18:46:10.879-05:00 INFO  [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 835.3 ms
2025-04-04T18:46:10.921-05:00 INFO  [ServerBootstrap] Graylog datanode 6.1.10+a308be3 starting up
2025-04-04T18:46:10.922-05:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.8.12-9-pve
2025-04-04T18:46:10.922-05:00 INFO  [ServerBootstrap] Deployment: deb
2025-04-04T18:46:10.922-05:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-04-04T18:46:10.923-05:00 INFO  [ServerBootstrap] Arch: amd64
2025-04-04T18:46:10.951-05:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-04-04T18:46:10.951-05:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-04-04T18:46:10.952-05:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-04-04T18:46:10.953-05:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-04-04T18:46:11.024-05:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-04-04T18:46:11.024-05:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-04-04T18:46:11.025-05:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-04-04T18:46:11.025-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-04-04T18:46:11.031-05:00 INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-04-04T18:46:11.033-05:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-04-04T18:46:11.035-05:00 INFO  [OpensearchProcessService] 

========================================================================================================
It seems you are starting Data node for the first time. The current configuration is not sufficient to
start the indexer process because a security configuration is missing. You have to either provide http
and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.
========================================================================================================

2025-04-04T18:46:11.036-05:00 INFO  [JerseyService] Starting Data node REST API
2025-04-04T18:46:11.036-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=76, OpensearchConfigurationService [RUNNING]=83}
2025-04-04T18:46:11.037-05:00 INFO  [ServerBootstrap] Graylog DataNode datanode up and running.
2025-04-04T18:46:11.381-05:00 INFO  [Version] HV000001: Hibernate Validator 8.0.1.Final
2025-04-04T18:46:11.560-05:00 INFO  [NetworkListener] Started listener bound to [0.0.0.0:8999]
2025-04-04T18:46:11.561-05:00 INFO  [HttpServer] [HttpServer] Started.
2025-04-04T18:46:11.562-05:00 INFO  [JerseyService] Started REST API at <0.0.0.0:8999>

The server.log post is posted in the next reply in this thread.

I know it says in the datanode.log:

You have to either provide http and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.

I’d go through the preflight website GUI but it won’t even load (browser saying it’s unable to connect)

How do I get it working again?

2

And this is server log:

/var/log/graylog-server/server.log 
2025-04-04T18:46:08.700-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[cloud_inputs=on, investigation_report_by_ai=on, data_tiering_cloud=off, composable_index_templates=off, preflight_web=on, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, configurable_value_units=on, new_report_creation=on, threat_coverage=on]}
2025-04-04T18:46:09.076-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 6.1.10+a308be3 [org.graylog.aws.AWSPlugin]
2025-04-04T18:46:09.077-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 6.1.10+a308be3 [org.graylog.integrations.IntegrationsPlugin]
2025-04-04T18:46:09.077-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 6.1.10+a308be3 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-04-04T18:46:09.078-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 6.1.10+a308be3 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-04-04T18:46:09.078-05:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 6.1.10+a308be3 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-04-04T18:46:09.090-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=deb
2025-04-04T18:46:09.213-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@59dc36d4, com.mongodb.Jep395RecordCodecProvider@12fcc71f, com.mongodb.KotlinCodecProvider@5679e96b]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.216-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@59dc36d4, com.mongodb.Jep395RecordCodecProvider@12fcc71f, com.mongodb.KotlinCodecProvider@5679e96b]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:09.233-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=16187996, minRoundTripTimeNanos=0}
2025-04-04T18:46:09.294-05:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.18
2025-04-04T18:46:11.143-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@59dc36d4, com.mongodb.Jep395RecordCodecProvider@12fcc71f, com.mongodb.KotlinCodecProvider@5679e96b]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:11.144-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.8.12-9-pve"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@59dc36d4, com.mongodb.Jep395RecordCodecProvider@12fcc71f, com.mongodb.KotlinCodecProvider@5679e96b]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-04-04T18:46:11.147-05:00 INFO  [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29998 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-04-04T18:46:11.163-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=21039597, minRoundTripTimeNanos=0}
2025-04-04T18:46:11.263-05:00 INFO  [IndexerDiscoveryProvider] No indexer hosts configured, using fallback http://127.0.0.1:9200
2025-04-04T18:46:11.471-05:00 INFO  [ServerBootstrap] Running 2 migrations...
2025-04-04T18:46:12.930-05:00 INFO  [IndexerDiscoveryProvider] No indexer hosts configured, using fallback http://127.0.0.1:9200
2025-04-04T18:46:12.988-05:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-04-04T18:46:12.989-05:00 INFO  [VersionProbe] Indexer is not available. Retry #1
2025-04-04T18:46:17.990-05:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-04-04T18:46:17.991-05:00 INFO  [VersionProbe] Indexer is not available. Retry #2

Only posting in this comment since it couldn’t fit in the original post for some reason.

3

Did you ever figure your problem out with this? I’m having the same issue between DataNode stuck at preflight prompt and Graylog not wanting to load.