Datanode not listening on 9200 - no opensearch

Graylog Central (peer support)
1

I’ve followed the Debian guide to install a standalone instance.

However, I see that my machine is not listening on 9200. The datanode service is started correctly and it listens on 8999. I’ve the impression that OpenSearch is not running at all.

Here are the logs

datanode.log 
2025-02-13T16:35:06.620+01:00 INFO  [ServerBootstrap] Graylog datanode 6.1.6+a644883 starting up
2025-02-13T16:35:06.620+01:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-31-cloud-amd64
2025-02-13T16:35:06.621+01:00 INFO  [ServerBootstrap] Deployment: deb
2025-02-13T16:35:06.621+01:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-13T16:35:06.622+01:00 INFO  [ServerBootstrap] Arch: amd64
2025-02-13T16:35:06.646+01:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-02-13T16:35:06.647+01:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-02-13T16:35:06.648+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-13T16:35:06.658+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-13T16:35:06.663+01:00 INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-02-13T16:35:06.664+01:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-02-13T16:35:06.667+01:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-13T16:35:06.670+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-13T16:35:06.676+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-13T16:35:06.701+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-13T16:35:06.702+01:00 INFO  [OpensearchProcessService] 

========================================================================================================
It seems you are starting Data node for the first time. The current configuration is not sufficient to
start the indexer process because a security configuration is missing. You have to either provide http
and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.
========================================================================================================

2025-02-13T16:35:06.734+01:00 INFO  [JerseyService] Starting Data node REST API
2025-02-13T16:35:06.740+01:00 INFO  [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=3, OpensearchProcessService [RUNNING]=4, OpensearchConfigurationService [RUNNING]=84, PeriodicalsService [RUNNING]=93}
2025-02-13T16:35:06.739+01:00 INFO  [CsrRequesterImpl] Triggered certificate signing request for this datanode
2025-02-13T16:35:06.740+01:00 INFO  [ServerBootstrap] Graylog DataNode datanode up and running.
2025-02-13T16:35:07.059+01:00 INFO  [Version] HV000001: Hibernate Validator 8.0.1.Final
2025-02-13T16:35:07.224+01:00 INFO  [NetworkListener] Started listener bound to [0.0.0.0:8999]
2025-02-13T16:35:07.225+01:00 INFO  [HttpServer] [HttpServer] Started.
2025-02-13T16:35:07.226+01:00 INFO  [JerseyService] Started REST API at <0.0.0.0:8999>

How could I troubleshoot this ?

The graylog-server service is properly installed, I’ve completed the preflights checks but then it fails to connect on the datanode port and I get his error in the logs (+WebUI not available)

2025-02-13T16:36:36.995+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to graylog.fqdn/172.28.14.4:9200. - Connection refused.
2025-02-13T16:36:36.995+01:00 INFO  [VersionProbe] Indexer is not available. Retry #134
2

I had the very same issue. The fix was to remove ‘elasticsearch_hosts’ line from server.conf. it would be useful for everyone if you could provide details of the conf files (after removing commented & empty lines via grep like grep -E '^(#|$)' -v server.conf

1 Like
3

Thanks for the suggestion.

However this settings was unset, and as per the docs

  • Default: https://127.0.0.1:9200 You may retain the default setting only if Elasticsearch is installed on the same host as the Graylog server.

Which I believe is my case.
Anyway, here are the 2 config files

server.conf 
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = REDACTED
root_password_sha2 = REDACTED
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
selfsigned_startup = true
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts
datanode.conf 
node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = REDACTED
root_password_sha2 = REDACTED
mongodb_uri = mongodb://localhost/graylog
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
1 Like
4

Hello,
I’d recommend removing content of the mongodb. Then commented out the selfsigned_startup configuration option and restart both graylog server and data node.

You should also check datanode logs, there will be most likely recorded a reason why the data node can’t start the opensearch process. Feel free to post your data node log here and I’ll try to suggest something else, if this won’t help.

Best regards,
Tomas

5

Implemented your suggestion.

On the pre-flight interface, the datanode (same system as graylog-server) is listed, but it still does not listen on 9200.

Here are its logs

datanode.log 
2025-02-14T09:28:28.647+01:00 INFO  [ServerBootstrap] Graylog datanode 6.1.6+a644883 starting up
2025-02-14T09:28:28.647+01:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-31-cloud-amd64
2025-02-14T09:28:28.648+01:00 INFO  [ServerBootstrap] Deployment: deb
2025-02-14T09:28:28.648+01:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-14T09:28:28.648+01:00 INFO  [ServerBootstrap] Arch: amd64
2025-02-14T09:28:28.701+01:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-02-14T09:28:28.701+01:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-02-14T09:28:28.701+01:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-14T09:28:28.784+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-14T09:28:28.794+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-14T09:28:28.795+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-14T09:28:28.801+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-14T09:28:28.802+01:00 INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-02-14T09:28:28.803+01:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-02-14T09:28:28.805+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-14T09:28:28.814+01:00 INFO  [OpensearchProcessService] 

========================================================================================================
It seems you are starting Data node for the first time. The current configuration is not sufficient to
start the indexer process because a security configuration is missing. You have to either provide http
and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.
========================================================================================================

2025-02-14T09:28:28.815+01:00 INFO  [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=1, OpensearchProcessService [RUNNING]=3, PeriodicalsService [RUNNING]=110, OpensearchConfigurationService [RUNNING]=112}
2025-02-14T09:28:28.815+01:00 INFO  [ServerBootstrap] Graylog DataNode datanode up and running.
2025-02-14T09:28:28.816+01:00 INFO  [JerseyService] Starting Data node REST API
2025-02-14T09:28:29.118+01:00 INFO  [Version] HV000001: Hibernate Validator 8.0.1.Final
2025-02-14T09:28:29.269+01:00 INFO  [NetworkListener] Started listener bound to [0.0.0.0:8999]
2025-02-14T09:28:29.270+01:00 INFO  [HttpServer] [HttpServer] Started.
2025-02-14T09:28:29.270+01:00 INFO  [JerseyService] Started REST API at <0.0.0.0:8999>
6

Good, that’s fine and expected. If you now look into your graylog-server logs, you’ll see that it’s telling you that the preflight configuration interface is waiting for you, on which address and what are the credentials.

If you open the preflight website, the wizard there will guide you through the process of creating/uploading a certificate authority and provisioning certificates for your data node.

Only when the data node is fully provisioned, it will automatically start the opensearch process and will listen on port 9200.

7

Thanks, indeed after fulfilling the preflight settings, the port 9200 is opened and the opensearch process is running.

Might it be something worth adding to the datanode configuration ?

8

However, it’s not possible to configure a graylog-server with SSL certificates (following this documentation) if the pre-flight is not performed.

When the graylog-server is configure to use TLS, it fails to start if there is no datanode available at 127.0.0.1:9200, which can only happen when the pre-flight is performed…

Here are the logs:

server.log 
2025-02-14T14:55:26.325+01:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[cloud_inputs=on, investigation_report_by_ai=on, data_tiering_cloud=off, composable_index_templates=off, preflight_web=on, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, configurable_value_units=on, new_report_creation=on, threat_coverage=on]}
2025-02-14T14:55:26.974+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 6.1.6+a644883 [org.graylog.aws.AWSPlugin]
2025-02-14T14:55:26.974+01:00 INFO  [CmdLineTool] Loaded plugin: Integrations 6.1.6+a644883 [org.graylog.integrations.IntegrationsPlugin]
2025-02-14T14:55:26.974+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 6.1.6+a644883 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-02-14T14:55:26.974+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 6.1.6+a644883 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-02-14T14:55:26.975+01:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 6.1.6+a644883 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-02-14T14:55:26.990+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=deb
2025-02-14T14:55:27.149+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-31-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@611a990b, com.mongodb.Jep395RecordCodecProvider@7cbeac65, com.mongodb.KotlinCodecProvider@7fe82967]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-14T14:55:27.167+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-31-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@611a990b, com.mongodb.Jep395RecordCodecProvider@7cbeac65, com.mongodb.KotlinCodecProvider@7fe82967]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-14T14:55:27.209+01:00 INFO  [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29995 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-02-14T14:55:27.210+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=30762701, minRoundTripTimeNanos=0}
2025-02-14T14:55:27.282+01:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.16
2025-02-14T14:55:27.733+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-31-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@611a990b, com.mongodb.Jep395RecordCodecProvider@7cbeac65, com.mongodb.KotlinCodecProvider@7fe82967]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-14T14:55:27.745+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-31-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@611a990b, com.mongodb.Jep395RecordCodecProvider@7cbeac65, com.mongodb.KotlinCodecProvider@7fe82967]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-14T14:55:27.746+01:00 INFO  [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-02-14T14:55:27.757+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5099684, minRoundTripTimeNanos=0}
2025-02-14T14:55:28.235+01:00 INFO  [ServerBootstrap] Running 2 migrations...
2025-02-14T14:55:28.454+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-02-14T14:55:28.477+01:00 INFO  [VersionProbe] Indexer is not available. Retry #1
2025-02-14T14:55:33.478+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.
2025-02-14T14:55:33.479+01:00 INFO  [VersionProbe] Indexer is not available. Retry #2
2025-02-14T14:55:38.480+01:00 ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to /127.0.0.1:9200. - Connection refused.

In my opinion this is a deadlock situation:

  1. Configure SSL certificate for accessing Graylog pre-flight config
  2. graylog-server fails to start because it expected a datanode on 127.0.0.1:9200
  3. the datanode’s opensearch on 9200 is not started because the pre-flight has not been done…

Am I missing something ?

9

just in case, in datanode.conf I would explicitly set: opensearch_http_port = 9200

10

NVM you have server + datanode installed at the same machine it seems.
Also. in /etc/mongodb.conf , you should set: bindIp: 0.0.0.0

11

I’ve set this, but this does not change anything, as OpenSearch is not starting.

I’ve tried specifying these options, hoping it would make opensearch start:

  • transport_certificate
  • transport_certificate_password
  • http_certificate
  • http_certificate_password

But it didn’t.
That .p12 file is correctly parsed, but the datanode complains that opensearch is not started:

2025-02-17T09:49:01.304+01:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
datanode.log 
2025-02-17T09:48:59.658+01:00 INFO  [OpensearchDataDirCompatibilityCheck] Found 0 indices and all of them are valid with current opensearch version 2.15.0
2025-02-17T09:48:59.875+01:00 INFO  [DatanodeDirectories] Opensearch of the node 74f09692-f70d-4c11-997d-669854d690c8 uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-02-17T09:49:00.927+01:00 INFO  [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 914.9 ms
2025-02-17T09:49:00.957+01:00 INFO  [ServerBootstrap] Graylog datanode 6.1.6+a644883 starting up
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-31-cloud-amd64
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] Deployment: deb
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-17T09:49:00.958+01:00 INFO  [ServerBootstrap] Arch: amd64
2025-02-17T09:49:00.979+01:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-02-17T09:49:00.979+01:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-02-17T09:49:00.980+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-17T09:49:00.985+01:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-17T09:49:00.990+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-17T09:49:01.000+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-17T09:49:01.006+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-17T09:49:01.077+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-17T09:49:01.250+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: graylog.<FQDN>, 172.28.14.4
2025-02-17T09:49:01.250+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following serial number: 622915052742269610821727451503118679977033641676
2025-02-17T09:49:01.251+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following validity: 2025-02-13T14:55:17.000+0100 - 2035-02-12T14:55:17.000+0100
2025-02-17T09:49:01.267+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: graylog.<FQDN>, 172.28.14.4
2025-02-17T09:49:01.268+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following serial number: 622915052742269610821727451503118679977033641676
2025-02-17T09:49:01.268+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following validity: 2025-02-13T14:55:17.000+0100 - 2035-02-12T14:55:17.000+0100
2025-02-17T09:49:01.270+01:00 INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-02-17T09:49:01.302+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2025-02-17T09:49:01.302+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
2025-02-17T09:49:01.303+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical].
2025-02-17T09:49:01.304+01:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[guava-33.3.1-jre.jar:?]
	at org.graylog.datanode.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:228) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.doRun(CmdLineTool.java:305) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.run(CmdLineTool.java:246) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
	Caused by: java.lang.NullPointerException: Cannot read the array length because "<parameter1>" is null
		at java.base/java.util.Arrays.stream(Unknown Source) ~[?:?]
		at org.graylog.datanode.configuration.TruststoreCreator.findRootCert(TruststoreCreator.java:101) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.TruststoreCreator.addRootCert(TruststoreCreator.java:65) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.variants.OpensearchSecurityConfiguration.configure(OpensearchSecurityConfiguration.java:92) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:159) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:218) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:95) ~[graylog-datanode.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[guava-33.3.1-jre.jar:?]
		at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.3.1-jre.jar:?]
		at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-17T09:49:01.312+01:00 INFO  [Server] SIGNAL received. Shutting down.
2025-02-17T09:49:01.314+01:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2025-02-17T09:49:01.314+01:00 INFO  [GracefulShutdown] Goodbye.
12

The temp credentials in the preflight were specifically added because of the understanding that you setup GL and the DataNode before you have secured GL itself via SSL. So that no prod passwords ever travel the network without TLS.
Can you try setting up GL/DataNode first and then add the SSL config later?

13

Well, I’m trying to set the SSL certificate at the start, because I want to avoid the pre-flight configuration.

I have seen docs for securing the WebUI access with certificate, but nothing is mentionned for the datanode SSL settings.

14

This error suggests that there is something wrong with the keystore used in the datanode. First of all I’d suggest updating your datanode package. Recently there were some changes implemented, which could help.

Could you please tell me what structure, keys and certificates does your .p12 keystore contain?

1 Like
15

I’m setting this up with Ansible → packages are up to date.

Here is the keystore contents:

graylog.p12 
keytool -list -v -keystore /etc/graylog/certs/graylog.p12 -storetype PKCS12

Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: graylog
Creation date: Feb 25, 2025
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=<GRAYLOG.FQDN>, O=Us, L=somewhere, C=US
Issuer: CN=<GRAYLOG.FQDN>, O=Us, L=somewhere, C=US
Serial number: <REDACTED>
Valid from: Sun Feb 23 16:57:55 CET 2025 until: Thu Feb 22 16:57:55 CET 2035
Certificate fingerprints:
	 SHA1: <REDACTED>	 SHA256: <REDACTED>
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen: no limit
]

#2: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
]

#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: graylog.backend
  IPAddress: 172.28.14.4
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: <REDACTED>
0010: <REDACTED>
]
]



*******************************************
*******************************************
16

Thank you!

I asked specifically about the update, because the method from the stacktrace:

TruststoreCreator.findRootCert

was removed in the latest 6.1.7 release (Released: 2025-02-17) and replaced with better certificate extraction from keystores. Updating the datanode to 6.1.7 could fix your problem.

17

It’s already in that latest version :wink:

package list 
dpkg -l | grep -E ".(elasticsearch|graylog|mongo)"
ii  graylog-6.1-repository           1-1                            all          Package to install Graylog 6.1 GPG key and repository
ii  graylog-datanode                 6.1.7-1                        amd64        Graylog data node
ii  graylog-server                   6.1.7-1                        amd64        Graylog server
ii  mongodb-database-tools           100.11.0                       amd64        mongodb-database-tools package provides tools for working with the MongoDB server: 
ii  mongodb-mongosh                  2.4.0                          amd64        MongoDB Shell CLI REPL Package
hi  mongodb-org                      8.0.5                          amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database             8.0.5                          amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database-tools-extra 8.0.5                          amd64        Extra MongoDB database tools
ii  mongodb-org-mongos               8.0.5                          amd64        MongoDB sharded cluster query router
ii  mongodb-org-server               8.0.5                          amd64        MongoDB database server
ii  mongodb-org-shell                8.0.5                          amd64        MongoDB shell client
ii  mongodb-org-tools                8.0.5                          amd64        MongoDB tools

I think that the content of the truststore is badly “formatted” or something.

What are the requirement of the certificates to create ?

18

Hi @3isenHeiM , you write that you already have the latest version (6.1.7) deployed. Can you please confirm that you still get the same Exception that @Tdvorak mentioned 3 posts above this one? Because your first mention of this exception is from Monday, 17.2. 9:27 in the morning while we released 6.1.7 the same day in the evening?

19

Well, per se I don’t find the error findRootCert anymore in the log files. But the datanode process exits with the error:

java.lang.NullPointerException: Cannot read the array length because "<parameter1>" is null

Here is the full log:

datanode.log 
2025-02-26T09:39:02.040+01:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-02-26T09:39:02.278+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms4g -Xmx4g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-02-26T09:39:02.406+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-30-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7569ea63, com.mongodb.Jep395RecordCodecProvider@ebd06a9, com.mongodb.KotlinCodecProvider@16c587de]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-26T09:39:02.413+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-30-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7569ea63, com.mongodb.Jep395RecordCodecProvider@ebd06a9, com.mongodb.KotlinCodecProvider@16c587de]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-26T09:39:02.454+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17937429, minRoundTripTimeNanos=0}
2025-02-26T09:39:02.457+01:00 INFO  [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29990 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-02-26T09:39:02.504+01:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 8.0.5
2025-02-26T09:39:02.642+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-30-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7569ea63, com.mongodb.Jep395RecordCodecProvider@ebd06a9, com.mongodb.KotlinCodecProvider@16c587de]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-26T09:39:02.644+01:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.2.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.1.0-30-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@7569ea63, com.mongodb.Jep395RecordCodecProvider@ebd06a9, com.mongodb.KotlinCodecProvider@16c587de]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-02-26T09:39:02.645+01:00 INFO  [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING}].
2025-02-26T09:39:02.647+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=25, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3344403, minRoundTripTimeNanos=0}
2025-02-26T09:39:02.677+01:00 INFO  [DatanodeDirectories] Opensearch of the node f34b798a-0a69-40ba-ba45-2b9980e564cf uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-02-26T09:39:02.703+01:00 INFO  [OpensearchConfigSync] Directory used for Opensearch process configuration is /var/lib/graylog-datanode/opensearch/config/opensearch
2025-02-26T09:39:02.716+01:00 INFO  [OpensearchConfigSync] Synchronizing Opensearch configuration
2025-02-26T09:39:02.719+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-02-26T09:39:02.720+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-02-26T09:39:02.720+01:00 INFO  [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-02-26T09:39:02.731+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-02-26T09:39:02.732+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-02-26T09:39:02.732+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-02-26T09:39:02.733+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-02-26T09:39:02.734+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-02-26T09:39:02.734+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-02-26T09:39:02.734+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-02-26T09:39:02.734+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-02-26T09:39:02.735+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-02-26T09:39:02.735+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-02-26T09:39:02.735+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-02-26T09:39:02.735+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/http-keystore.p12
2025-02-26T09:39:02.736+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/datanode-truststore.p12
2025-02-26T09:39:02.736+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/transport-keystore.p12
2025-02-26T09:39:02.737+01:00 INFO  [FullDirSync] Deleting obsolete directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-02-26T09:39:02.737+01:00 INFO  [FullDirSync] Deleting obsolete file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-02-26T09:39:02.739+01:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch
2025-02-26T09:39:02.741+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/log4j2.properties
2025-02-26T09:39:02.742+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/jvm.options
2025-02-26T09:39:02.742+01:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability
2025-02-26T09:39:02.743+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-observability/observability.yml
2025-02-26T09:39:02.744+01:00 INFO  [FullDirSync] Synchronizing directory /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security
2025-02-26T09:39:02.744+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/whitelist.yml
2025-02-26T09:39:02.745+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/opensearch.yml.example
2025-02-26T09:39:02.745+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/allowlist.yml
2025-02-26T09:39:02.745+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles_mapping.yml
2025-02-26T09:39:02.746+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/internal_users.yml
2025-02-26T09:39:02.746+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/action_groups.yml
2025-02-26T09:39:02.746+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/audit.yml
2025-02-26T09:39:02.747+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/config.yml
2025-02-26T09:39:02.747+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/tenants.yml
2025-02-26T09:39:02.748+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/nodes_dn.yml
2025-02-26T09:39:02.748+01:00 INFO  [FullDirSync] Synchronizing file /var/lib/graylog-datanode/opensearch/config/opensearch/opensearch-security/roles.yml
2025-02-26T09:39:02.750+01:00 INFO  [OpensearchDistributionProvider] Found following opensearch distributions: [/usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64]
2025-02-26T09:39:02.763+01:00 INFO  [OpensearchDataDirCompatibilityCheck] Found 0 indices and all of them are valid with current opensearch version 2.15.0
2025-02-26T09:39:02.939+01:00 INFO  [DatanodeDirectories] Opensearch of the node f34b798a-0a69-40ba-ba45-2b9980e564cf uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/log/graylog-datanode/opensearch', configurationSourceDir='Optional[/etc/graylog/datanode]', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config', opensearchProcessConfigurationDir='/var/lib/graylog-datanode/opensearch/config/opensearch'}
2025-02-26T09:39:03.877+01:00 INFO  [DbEntitiesScanner] 16 entities have been scanned and added to DB Entity Catalog, it took 810.7 ms
2025-02-26T09:39:03.920+01:00 INFO  [ServerBootstrap] Graylog datanode 6.1.7+6ec0bac starting up
2025-02-26T09:39:03.924+01:00 INFO  [ServerBootstrap] JRE: Eclipse Adoptium 17.0.14 on Linux 6.1.0-30-cloud-amd64
2025-02-26T09:39:03.925+01:00 INFO  [ServerBootstrap] Deployment: deb
2025-02-26T09:39:03.925+01:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 12 (bookworm) (debian)
2025-02-26T09:39:03.925+01:00 INFO  [ServerBootstrap] Arch: amd64
2025-02-26T09:39:03.949+01:00 INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-02-26T09:39:03.949+01:00 INFO  [PeriodicalsService] Delaying start of 1 periodicals until this node becomes leader ...
2025-02-26T09:39:04.040+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.NodePingPeriodical] periodical in [0s], polling every [1s].
2025-02-26T09:39:04.048+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-02-26T09:39:04.064+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [1800s].
2025-02-26T09:39:04.068+01:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-02-26T09:39:04.089+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-02-26T09:39:04.098+01:00 INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-02-26T09:39:04.278+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following alternative names: graylog.<redacted>, 172.28.14.4
2025-02-26T09:39:04.279+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following serial number: 429255647533457603859532427351443426371580009640
2025-02-26T09:39:04.280+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch transport certificate has following validity: 2025-02-25T09:21:04.000+0100 - 2035-02-24T09:21:04.000+0100
2025-02-26T09:39:04.299+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following alternative names: graylog.<redacted>, 172.28.14.4
2025-02-26T09:39:04.300+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following serial number: 429255647533457603859532427351443426371580009640
2025-02-26T09:39:04.300+01:00 INFO  [OpensearchSecurityConfiguration] Opensearch HTTP certificate has following validity: 2025-02-25T09:21:04.000+0100 - 2035-02-24T09:21:04.000+0100
2025-02-26T09:39:04.303+01:00 INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-02-26T09:39:04.324+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical].
2025-02-26T09:39:04.325+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical].
2025-02-26T09:39:04.325+01:00 INFO  [Periodicals] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2025-02-26T09:39:04.326+01:00 ERROR [ServerBootstrap] Graylog DataNode startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[OpensearchConfigurationService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:772) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:584) ~[guava-33.3.1-jre.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:298) ~[guava-33.3.1-jre.jar:?]
	at org.graylog.datanode.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:228) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.doRun(CmdLineTool.java:305) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.CmdLineTool.run(CmdLineTool.java:246) [graylog-datanode.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: OpensearchConfigurationService [FAILED]
	Caused by: java.lang.NullPointerException: Cannot read the array length because "<parameter1>" is null
		at java.base/java.util.Arrays.stream(Unknown Source) ~[?:?]
		at org.graylog2.security.TruststoreCreator.toX509Certs(TruststoreCreator.java:80) ~[graylog2-server-6.1.7.jar:?]
		at org.graylog2.security.TruststoreCreator.addFromKeystore(TruststoreCreator.java:74) ~[graylog2-server-6.1.7.jar:?]
		at org.graylog.datanode.configuration.variants.OpensearchSecurityConfiguration.configure(OpensearchSecurityConfiguration.java:92) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.get(OpensearchConfigurationService.java:159) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.triggerConfigurationChangedEvent(OpensearchConfigurationService.java:218) ~[graylog-datanode.jar:?]
		at org.graylog.datanode.configuration.OpensearchConfigurationService.startUp(OpensearchConfigurationService.java:95) ~[graylog-datanode.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:64) ~[guava-33.3.1-jre.jar:?]
		at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.3.1-jre.jar:?]
		at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2025-02-26T09:39:04.333+01:00 INFO  [Server] SIGNAL received. Shutting down.
2025-02-26T09:39:04.334+01:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2025-02-26T09:39:04.334+01:00 INFO  [GracefulShutdown] Goodbye.

And here is the datanode config:

node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = REDACTED
root_password_sha2 = REDACTED
mongodb_uri = mongodb://localhost/graylog
bind_address = 127.0.0.1
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
transport_certificate = /etc/graylog/certs/graylog.p12
transport_certificate_password = changeit
http_certificate = /etc/graylog/certs/graylog.p12
http_certificate_password = changeit

The contents of the keystore are the same than posted here.

20

can you try with a new/recreated keystore where you use “datanode” instead of “graylog” as the alias for the key?

1 Like