Preflight: Unconfigured datanode appears once in list, then disappears

Graylog Central (peer support)
1

1. Describe your incident:
I’m trying to configure Graylog to run on the Railway hosting platform. I have Mongo connected and working and I have communication between the containers working (confirmed by pulling http://datanode:8999 with curl on the graynode container).

During preflight configuration, the datanode appears ONCE in the list of discovered nodes, then permanently disappears (goes back to “No data nodes have been found.”) It can be made to appear briefly by wiping its configuration, removing its Mongo entry, and restarting.

During the moment when the datanode is visible in the preflight list, Chrome shows this response from the /api/data_nodes endpoint:

[
    {
        "node_id": "e5325ad9-9355-40ef-b558-802c140a795c",
        "transport_address": "http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/",
        "status": "STARTING",
        "error_msg": null,
        "hostname": "18d9c9744dd4",
        "short_node_id": "e5325ad9",
        "data_node_status": "STARTING"
    }
]

But on the next endpoint poll, this reverts to an empty list [] and it is no longer possible to continue setup in the frontend. The graylog container does not print any information to the console during this.

2. Describe your environment:

  • OS Information:
    Docker on Railway.app

  • Package Version:
    Official graylog-6.2.1 and graylog-datanode-6.2.1 Docker images

  • Service logs, configurations, and environment variables:

Datanode:

GRAYLOG_DATANODE_NODE_ID_FILE="/var/lib/graylog-datanode/node-id"
GRAYLOG_DATANODE_MONGODB_URI="${{MongoDB.MONGO_URL}}/graylog"
GRAYLOG_DATANODE_PASSWORD_SECRET="redacted"
GRAYLOG_DATANODE_SKIP_PREFLIGHT_CHECKS="true"
GRAYLOG_DATANODE_BIND_ADDRESS="::"
  • Volume mounted at /var/lib/graylog-datanode
  • Preflight checks are disabled because mmap settings are not changeable on the container platform

Graylog:

GRAYLOG_MONGODB_URI="${{MongoDB.MONGO_URL}}/graylog"
GRAYLOG_PASSWORD_SECRET="redacted"
GRAYLOG_ROOT_PASSWORD_SHA2="redacted"
GRAYLOG_HTTP_BIND_ADDRESS="0.0.0.0:9000"
GRAYLOG_HTTP_EXTERNAL_URI="https://logs.redacted.net/"
GRAYLOG_ELASTICSEARCH_DISCOVERY_ENABLED="true"
  • Volume mounted at /usr/share/graylog/data/data

Mongo:
Using mongo:7 official container

MONGO_INITDB_ROOT_PASSWORD="redacted"
MONGO_INITDB_ROOT_USERNAME="mongo"
MONGO_PUBLIC_URL="mongodb://${{MONGO_INITDB_ROOT_USERNAME}}:${{MONGO_INITDB_ROOT_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}"
MONGO_URL="mongodb://${{MONGO_INITDB_ROOT_USERNAME}}:${{MONGO_INITDB_ROOT_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:27017"
MONGOHOST="${{RAILWAY_PRIVATE_DOMAIN}}"
MONGOPASSWORD="${{MONGO_INITDB_ROOT_PASSWORD}}"
MONGOPORT="27017"
MONGOUSER="${{MONGO_INITDB_ROOT_USERNAME}}"

3. What steps have you already taken to try and solve the problem?

  • Configured datanode bind address to ::
  • Confirmed Mongo connections working on graylog and datanode containers (datanode identity and provisioning certificate are both written to Mongo)
  • Wiped volumes and Mongo collections several times to restart from scratch
  • Confirmed that http://datanode:8999 is accessible over ipv6, using curl in the graylog container
  • Tried configuring the environment variable GRAYLOG_ELASTICSEARCH_HOSTS="http://datanode:9200" - this causes graylog to never serve the preflight configuration page, instead it prints retry messages forever.
  • Tried configuring <Logger name="org.graylog2" level="debug"/> in /usr/share/graylog/data/config/log4j2.xml and restarting graylog with SIGHUP within the running container - but this does not add any messages at all, it still logs at INFO level as far as I can tell.

4. How can the community help?

  • Are datanodes connectable ipv6-only? my run platform does not support ipv4 communication between containers.
  • The datanode registers itself in mongo datanodes collection with this document. does this look correct, especially the last portion of the ipv6 address?
{
  "_id": "68377f45428d56f22a24f397",
  "node_id": "e5325ad9-9355-40ef-b558-802c140a795c",
  "datanode_status": "STARTING",
  "hostname": "18d9c9744dd4",
  "is_leader": false,
  "last_seen": {
    "$timestamp": "7509610837993062401"
  },
  "transport_address": "http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/"
}
  • Why does the datanode appear and then disappear from the setup list? How can I continue setup?
  • Is there anything wrong with the response from the datanode on port 8999 with curl, captured from inside the graylog container?
# curl http://datanode:8999
{"operating_system":{"os_name":"Linux","os_version":"6.12.12+bpo-cloud-amd64","java_version":"17.0.14","user_name":"graylog"},"opensearch":{"opensearch_version":"2.15.0","node":{"node_name":"18d9c9744dd4","state":"WAITING_FOR_CONFIGURATION","rest_base_url":"","process":{"pid":-1,"alive":false,"started":null}}},"datanode_directories":{"data_target_dir":"file:///var/lib/graylog-datanode/opensearch/data/","logs_target_dir":"file:///var/lib/graylog-datanode/opensearch/logs/","configuration_source_dir":null,"configuration_target_dir":"file:///var/lib/graylog-datanode/opensearch/config/"},"dto":{"status":"STARTING","error_msg":null,"cert_valid_until":null,"data_node_status":"STARTING","cert_valid_until":null,"cluster_address":null,"rest_api_address":null,"action_queue":null,"datanode_version":null,"opensearch_roles":null,"configuration_warnings":null,"version_compatible":false,"id":"e5325ad9-9355-40ef-b558-802c140a795c","object_id":"68377f45428d56f22a24f397","node_id":"e5325ad9-9355-40ef-b558-802c140a795c","is_leader":false,"short_node_id":"e5325ad9","transport_address":"http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/","hostname":"18d9c9744dd4","last_seen":"2025-05-28T21:25:25.000Z","is_master":false},"data_node_version":"6.2.1+f599f1d"}
3
Graylog output 
Starting Container
Stopping Container
Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/356cf24c-c504-4235-943a-812bc3b62491/vol_e64mm982uby1wthk
Starting Container
2025-05-29 21:01:37,252 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[show_security_events_in_pedt=off, data_tiering_cloud=off, preflight_web=on, configurable_value_units=on, setup_mode=on, cloud_inputs=on, investigation_report_by_ai=on, show_executive_dashboard_page=off, composable_index_templates=off, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, data_warehouse_search=on, new_report_creation=on, threat_coverage=on]}
2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 6.2.2+13ba949 [org.graylog.aws.AWSPlugin]
2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Integrations 6.2.2+13ba949 [org.graylog.integrations.IntegrationsPlugin]
2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 6.2.2+13ba949 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-05-29 21:01:37,645 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: OpenSearch 2 Support 6.2.2+13ba949 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-05-29 21:01:37,645 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 7 Support 6.2.2+13ba949 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-05-29 21:01:37,656 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2025-05-29 21:01:37,788 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29 21:01:37,816 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 3. Remaining time: 29993 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}].
2025-05-29 21:01:37,790 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29 21:01:37,913 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=16435239, minRoundTripTimeNanos=0}
2025-05-29 21:01:38,047 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - Connected to MongoDB version 7.0.20
2025-05-29 21:01:39,076 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29 21:01:39,077 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1657403, minRoundTripTimeNanos=0}
2025-05-29 21:01:39,077 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29 21:01:39,078 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 12. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}].
2025-05-29 21:01:39,165 INFO : org.graylog2.plugin.system.FilePersistedNodeIdProvider - No node ID file found. Generated: ae83ff81-997b-49da-be53-a119b897458f
2025-05-29 21:01:39,233 INFO : org.graylog2.configuration.IndexerDiscoveryProvider - No indexer hosts configured, using fallback http://127.0.0.1:9200
2025-05-29 21:01:39,529 INFO : org.graylog2.bootstrap.ServerBootstrap - Running 2 migrations...
2025-05-29 21:01:39,549 INFO : org.graylog2.bootstrap.ServerBootstrap - Fresh installation detected, starting configuration webserver
2025-05-29 21:01:39,555 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2025-05-29 21:01:39,552 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 3 periodicals ...
2025-05-29 21:01:39,554 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical] periodical in [2s], polling every [2s].
2025-05-29 21:01:39,555 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-05-29 21:01:39,925 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2025-05-29 21:01:40,125 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2025-05-29 21:01:40,126 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2025-05-29 21:01:40,129 INFO : org.graylog2.bootstrap.preflight.PreflightJerseyService - 
                                                             ---
                                                             ---
                                                             ---
    ########  ###   ######### ##########   ####         #### ---         .----               ----
  ###############   ###################### #####       ####  ---      ------------       .----------- --
 #####     ######   #####              #### ####      ####   ---     ---        ---     ---        -----
  #############                                ####                                     -----     ----
     ######                                   ####                                          -------
####         ####   ####       ############  ####     ####   ---    --           ---   ---           ---

###           ###   ####     ##############   ####   ####    ---   ---            --   --             --
========================================================================================================
####         ####   ####    ####       ####    #### ####     ---   ---            --   --            .--

#####       #####   ####    ####       ####     #######      ---    ---          ---   ---           ---
It seems you are starting Graylog for the first time. To set up a fresh install, a setup interface has
 ################   ####     ##############     ######-       --     ----      ----      ---       -----
been started. You must log in to it to perform the initial configuration and continue.
   ##############   ####      #############      #####        -----   -----------         ----------  --

Initial configuration is accessible at 0.0.0.0:9000, with username 'admin' and password 'redacted'.
             ####                                ####                                                ---
Try clicking on http://admin:redacted@0.0.0.0:9000
#####       ####                                ####                                     -          .--

========================================================================================================
4
Datanode output 
Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/6f17995f-18d1-43db-8771-468e7554d232/vol_1h8ozzf79g74m0ul
Starting Container
2025-05-29T21:13:21.627Z WARN  [CmdLineTool] Couldn't create native lib dir </native_libs>. Unable to set ZstdTempFolder system property.
java.nio.file.AccessDeniedException: /native_libs
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
2025-05-29T21:13:21.755Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-05-29T21:13:21.760Z WARN  [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-05-29T21:13:22.137Z INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-05-29T21:13:22.138Z INFO  [DatanodeBootstrap] Skipping preflight checks
	at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:162) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:184) ~[graylog-datanode.jar:?]
	... 7 more
Caused by: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]
	at java.base/java.io.FileInputStream.open(Unknown Source) ~[?:?]
	at java.base/java.io.FileInputStream.<init>(Unknown Source) ~[?:?]
	at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:157) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:184) ~[graylog-datanode.jar:?]
	... 7 more
2025-05-29T21:13:23.369Z INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s].
2025-05-29T21:13:23.370Z INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s].
2025-05-29T21:13:23.370Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2025-05-29T21:13:23.371Z INFO  [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s].
2025-05-29T21:13:23.414Z INFO  [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration.
2025-05-29T21:13:23.420Z INFO  [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12
2025-05-29T21:13:23.440Z INFO  [OpensearchProcessService] 

========================================================================================================
It seems you are starting Data node for the first time. The current configuration is not sufficient to
start the indexer process because a security configuration is missing. You have to either provide http
and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely.
========================================================================================================

2025-05-29T21:13:23.441Z INFO  [JerseyService] Starting Data node REST API
2025-05-29T21:13:23.441Z INFO  [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=11, OpensearchConfigurationService [RUNNING]=76}
2025-05-29T21:13:23.442Z INFO  [DatanodeBootstrap] Graylog DataNode datanode up and running.
2025-05-29T21:13:24.017Z INFO  [Version] HV000001: Hibernate Validator 8.0.2.Final
2025-05-29T21:13:24.293Z INFO  [NetworkListener] Started listener bound to [[::]:8999]
2025-05-29T21:13:22.716Z INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3850e90c, com.mongodb.Jep395RecordCodecProvider@3d9f5016, com.mongodb.KotlinCodecProvider@7e91ed74]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29T21:13:24.295Z INFO  [HttpServer] [HttpServer] Started.
2025-05-29T21:13:24.296Z INFO  [JerseyService] Started REST API at <:::8999>
2025-05-29T21:13:22.720Z INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3850e90c, com.mongodb.Jep395RecordCodecProvider@3d9f5016, com.mongodb.KotlinCodecProvider@7e91ed74]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-05-29T21:13:22.751Z INFO  [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29991 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}].
2025-05-29T21:13:22.789Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17555410, minRoundTripTimeNanos=0}
2025-05-29T21:13:22.901Z INFO  [DatanodeDirectories] Opensearch of the node e5325ad9-9355-40ef-b558-802c140a795c uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/lib/graylog-datanode/opensearch/logs', configurationSourceDir='Optional.empty', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-05-29T21:13:23.324Z INFO  [DatanodeBootstrap] Graylog Data Node 6.2.2+13ba949 starting up (command: datanode)
2025-05-29T21:13:23.325Z INFO  [DatanodeBootstrap] JRE: Eclipse Adoptium 17.0.15 on Linux 6.12.12+bpo-cloud-amd64
2025-05-29T21:13:23.325Z INFO  [DatanodeBootstrap] Deployment: container
2025-05-29T21:13:23.325Z INFO  [DatanodeBootstrap] OS: Ubuntu 22.04.5 LTS (jammy)
2025-05-29T21:13:23.326Z INFO  [DatanodeBootstrap] Arch: amd64
2025-05-29T21:13:23.362Z INFO  [PeriodicalsService] Starting 6 periodicals ...
2025-05-29T21:13:23.363Z INFO  [PeriodicalsService] Delaying start of 0 periodicals until this node becomes leader ...
2025-05-29T21:13:23.364Z INFO  [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s].
2025-05-29T21:13:23.368Z ERROR [PeriodicalsService] Could not initialize periodical.
java.lang.RuntimeException: org.graylog.datanode.configuration.DatanodeKeystoreException: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory)
	at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:192) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.periodicals.NodePingPeriodical.registerServer(NodePingPeriodical.java:163) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.periodicals.NodePingPeriodical.initialize(NodePingPeriodical.java:134) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.initializers.PeriodicalsService.startPeriodicals(PeriodicalsService.java:76) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.initializers.PeriodicalsService.startUp(PeriodicalsService.java:61) ~[graylog-datanode.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:63) ~[guava-33.4.7-jre.jar:?]
	at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104) ~[guava-33.4.7-jre.jar:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog.datanode.configuration.DatanodeKeystoreException: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory)
5

Hey @syntaxaire,
By skipping the preflight checks you are also skipping a part of the code that initially creates the datanode keystore. Then datanode fails when trying to persist its certificate.

Is there any reason why you are disabling preflight checks? I’d suggest enabling them again and start the preflight/provisioning. Then everything should be good.

6

I have preflight checks disabled since is the only way I can run the datanode. Otherwise:

Exception in thread "main" org.graylog2.bootstrap.preflight.PreflightCheckException: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl)
	at org.graylog.datanode.bootstrap.preflight.OpenSearchPreconditionsCheck.runCheck(OpenSearchPreconditionsCheck.java:51)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421)
	at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52)
	at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:82)
	at org.graylog.datanode.bootstrap.DatanodeBootstrap.beforeInjectorCreation(DatanodeBootstrap.java:70)
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358)
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283)
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59)

It seems the solution is to set node.store.allow_mmap: "false" in elasticsearch.yml. Since the GRAYLOG_DATANODE environment variables are not documented I’m not sure if there is a variable for this, or whether I need to create a custom datanode image to apply the change.

(Edit: mmap is not configurable on Railway)

7

Ah, I see, thanks! You should be able to configure this in Data Node Configuration Overrides .

Then you can probably re-enable the checks.

8

I configured it like so but the preflight mmap check still fails.

I set a custom start command to avoid making an entire new image:

/bin/sh -c "cd /etc/graylog/datanode &&
 echo \"node.store.allow_mmap=false\" >> overrideconfig.conf &&
 chown graylog: overrideconfig.conf &&
 echo \"\\nopensearch_configuration_overrides_file = /etc/graylog/datanode/overrideconfig.conf\" >> datanode.conf &&
 ls -l &&
 cat datanode.conf &&
 cat overrideconfig.conf &&
 cd &&
 /entrypoint.sh graylog-datanode"

See the output of the ls and cat commands at the beginning of the deployment log:

Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/74b021d7-bc76-4a5b-8e78-ed0a50063931/vol_1h8ozzf79g74m0ul
total 16
-rw-r--r-- 1 graylog graylog  850 Jun  4 18:56 datanode.conf
-rw-r--r-- 1 graylog graylog  444 May  7 11:18 jvm.options
-rw-r--r-- 1 graylog graylog 1500 May  7 11:18 log4j2.xml
-rw-r--r-- 1 graylog graylog  280 Jun  4 18:56 overrideconfig.conf
 
opensearch_configuration_overrides_file = /etc/graylog/datanode/overrideconfig.conf
node.store.allow_mmap=false
2025-06-04T16:09:16.122Z WARN  [CmdLineTool] Couldn't create native lib dir </native_libs>. Unable to set ZstdTempFolder system property.
java.nio.file.AccessDeniedException: /native_libs
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
2025-06-04T16:09:16.230Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-06-04T16:09:16.234Z WARN  [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-06-04T16:09:16.569Z INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-06-04T16:09:16.950Z INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-04T16:09:17.093Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17689048, minRoundTripTimeNanos=0}
2025-06-04T16:09:16.953Z INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-04T16:09:16.983Z INFO  [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29991 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}].
2025-06-04T16:09:17.189Z INFO  [DatanodeDirectories] Opensearch of the node e5325ad9-9355-40ef-b558-802c140a795c uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/lib/graylog-datanode/opensearch/logs', configurationSourceDir='Optional.empty', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'}
2025-06-04T16:09:22.370Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1862446, minRoundTripTimeNanos=0}
2025-06-04T16:09:17.235Z INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-06-04T16:09:22.370Z INFO  [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}].
2025-06-04T16:09:22.411Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 7.0.20
2025-06-04T16:09:22.421Z ERROR [PreflightCheckService] Preflight check failed with error: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl)
2025-06-04T16:09:22.421Z ERROR [CmdLineTool] Startup error:
org.graylog2.bootstrap.preflight.PreflightCheckException: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl)
	at org.graylog.datanode.bootstrap.preflight.OpenSearchPreconditionsCheck.runCheck(OpenSearchPreconditionsCheck.java:51) ~[graylog-datanode.jar:?]
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421) ~[guava-33.4.7-jre.jar:?]
	at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52) ~[graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:82) ~[graylog-datanode.jar:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?]
	at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?]
	at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?]
2025-06-04T16:09:21.415Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-06-04T16:09:21.419Z WARN  [PluginLoader] Plugin directory /plugin does not exist, not loading plugins.
2025-06-04T16:09:21.752Z INFO  [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
9

Thanks, you are right. I just replicated the problem locally. The failing check is already on the datanode side. There is no way how to skip it without also skipping the keystore check as well.

The only way how to fix that is a small change in the datanode code. I prepared something, will discuss the problem and solution with my colleagues and will let you know what happens next.

1 Like
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.