# Preflight: Unconfigured datanode appears once in list, then disappears **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [May 29, 2025, 2:20pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760 "2025-05-29T14:20:33Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![syntaxaire](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@syntaxaire](https://community.graylog.org/u/syntaxaire)\ **Post date:** [May 29, 2025, 2:20pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/1 "2025-05-29T14:20:33Z")
**1. Describe your incident:** I’m trying to configure Graylog to run on the Railway hosting platform. I have Mongo connected and working and I have communication between the containers working (confirmed by pulling `http://datanode:8999` with `curl` on the graynode container). During preflight configuration, the datanode appears ONCE in the list of discovered nodes, then permanently disappears (goes back to “No data nodes have been found.”) It can be made to appear briefly by wiping its configuration, removing its Mongo entry, and restarting. During the moment when the datanode is visible in the preflight list, Chrome shows this response from the `/api/data_nodes` endpoint: ```auto [ { "node_id": "e5325ad9-9355-40ef-b558-802c140a795c", "transport_address": "http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/", "status": "STARTING", "error_msg": null, "hostname": "18d9c9744dd4", "short_node_id": "e5325ad9", "data_node_status": "STARTING" } ] ``` But on the next endpoint poll, this reverts to an empty list `[]` and it is no longer possible to continue setup in the frontend. The graylog container does not print any information to the console during this. **2. Describe your environment:** - OS Information: Docker on Railway.app - Package Version: Official graylog-6.2.1 and graylog-datanode-6.2.1 Docker images - Service logs, configurations, and environment variables: **Datanode:** ```auto GRAYLOG_DATANODE_NODE_ID_FILE="/var/lib/graylog-datanode/node-id" GRAYLOG_DATANODE_MONGODB_URI="${{MongoDB.MONGO_URL}}/graylog" GRAYLOG_DATANODE_PASSWORD_SECRET="redacted" GRAYLOG_DATANODE_SKIP_PREFLIGHT_CHECKS="true" GRAYLOG_DATANODE_BIND_ADDRESS="::" ``` - Volume mounted at `/var/lib/graylog-datanode` - Preflight checks are disabled because mmap settings are not changeable on the container platform **Graylog:** ```auto GRAYLOG_MONGODB_URI="${{MongoDB.MONGO_URL}}/graylog" GRAYLOG_PASSWORD_SECRET="redacted" GRAYLOG_ROOT_PASSWORD_SHA2="redacted" GRAYLOG_HTTP_BIND_ADDRESS="0.0.0.0:9000" GRAYLOG_HTTP_EXTERNAL_URI="https://logs.redacted.net/" GRAYLOG_ELASTICSEARCH_DISCOVERY_ENABLED="true" ``` - Volume mounted at `/usr/share/graylog/data/data` **Mongo:** Using `mongo:7` official container ```auto MONGO_INITDB_ROOT_PASSWORD="redacted" MONGO_INITDB_ROOT_USERNAME="mongo" MONGO_PUBLIC_URL="mongodb://${{MONGO_INITDB_ROOT_USERNAME}}:${{MONGO_INITDB_ROOT_PASSWORD}}@${{RAILWAY_TCP_PROXY_DOMAIN}}:${{RAILWAY_TCP_PROXY_PORT}}" MONGO_URL="mongodb://${{MONGO_INITDB_ROOT_USERNAME}}:${{MONGO_INITDB_ROOT_PASSWORD}}@${{RAILWAY_PRIVATE_DOMAIN}}:27017" MONGOHOST="${{RAILWAY_PRIVATE_DOMAIN}}" MONGOPASSWORD="${{MONGO_INITDB_ROOT_PASSWORD}}" MONGOPORT="27017" MONGOUSER="${{MONGO_INITDB_ROOT_USERNAME}}" ``` **3. What steps have you already taken to try and solve the problem?** - Configured datanode bind address to :: - Confirmed Mongo connections working on graylog and datanode containers (datanode identity and provisioning certificate are both written to Mongo) - Wiped volumes and Mongo collections several times to restart from scratch - Confirmed that [http://datanode:8999](http://datanode:8999) is accessible over ipv6, using `curl` in the graylog container - Tried configuring the environment variable `GRAYLOG_ELASTICSEARCH_HOSTS="http://datanode:9200"` - this causes graylog to never serve the preflight configuration page, instead it prints retry messages forever. - Tried configuring `` in `/usr/share/graylog/data/config/log4j2.xml` and restarting graylog with SIGHUP within the running container - but this does not add any messages at all, it still logs at INFO level as far as I can tell. **4. How can the community help?** - Are datanodes connectable ipv6-only? my run platform does not support ipv4 communication between containers. - The datanode registers itself in mongo `datanodes` collection with this document. does this look correct, especially the last portion of the ipv6 address? ```auto { "_id": "68377f45428d56f22a24f397", "node_id": "e5325ad9-9355-40ef-b558-802c140a795c", "datanode_status": "STARTING", "hostname": "18d9c9744dd4", "is_leader": false, "last_seen": { "$timestamp": "7509610837993062401" }, "transport_address": "http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/" } ``` - Why does the datanode appear and then disappear from the setup list? How can I continue setup? - Is there anything wrong with the response from the datanode on port 8999 with `curl`, captured from inside the graylog container? ```auto # curl http://datanode:8999 {"operating_system":{"os_name":"Linux","os_version":"6.12.12+bpo-cloud-amd64","java_version":"17.0.14","user_name":"graylog"},"opensearch":{"opensearch_version":"2.15.0","node":{"node_name":"18d9c9744dd4","state":"WAITING_FOR_CONFIGURATION","rest_base_url":"","process":{"pid":-1,"alive":false,"started":null}}},"datanode_directories":{"data_target_dir":"file:///var/lib/graylog-datanode/opensearch/data/","logs_target_dir":"file:///var/lib/graylog-datanode/opensearch/logs/","configuration_source_dir":null,"configuration_target_dir":"file:///var/lib/graylog-datanode/opensearch/config/"},"dto":{"status":"STARTING","error_msg":null,"cert_valid_until":null,"data_node_status":"STARTING","cert_valid_until":null,"cluster_address":null,"rest_api_address":null,"action_queue":null,"datanode_version":null,"opensearch_roles":null,"configuration_warnings":null,"version_compatible":false,"id":"e5325ad9-9355-40ef-b558-802c140a795c","object_id":"68377f45428d56f22a24f397","node_id":"e5325ad9-9355-40ef-b558-802c140a795c","is_leader":false,"short_node_id":"e5325ad9","transport_address":"http://[fd12:a156:788a:0:a000:13:91b2:a064%railnet0]:8999/","hostname":"18d9c9744dd4","last_seen":"2025-05-28T21:25:25.000Z","is_master":false},"data_node_version":"6.2.1+f599f1d"} ``` ---
**Author:** ![syntaxaire](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@syntaxaire](https://community.graylog.org/u/syntaxaire)\ **Post date:** [May 29, 2025, 9:05pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/3 "2025-05-29T21:05:55Z")
> **Graylog output** > > ```auto > Starting Container > Stopping Container > Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/356cf24c-c504-4235-943a-812bc3b62491/vol_e64mm982uby1wthk > Starting Container > 2025-05-29 21:01:37,252 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[show_security_events_in_pedt=off, data_tiering_cloud=off, preflight_web=on, configurable_value_units=on, setup_mode=on, cloud_inputs=on, investigation_report_by_ai=on, show_executive_dashboard_page=off, composable_index_templates=off, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, data_warehouse_search=on, new_report_creation=on, threat_coverage=on]} > 2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 6.2.2+13ba949 [org.graylog.aws.AWSPlugin] > 2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Integrations 6.2.2+13ba949 [org.graylog.integrations.IntegrationsPlugin] > 2025-05-29 21:01:37,644 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 6.2.2+13ba949 [org.graylog.plugins.threatintel.ThreatIntelPlugin] > 2025-05-29 21:01:37,645 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: OpenSearch 2 Support 6.2.2+13ba949 [org.graylog.storage.opensearch2.OpenSearch2Plugin] > 2025-05-29 21:01:37,645 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 7 Support 6.2.2+13ba949 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin] > 2025-05-29 21:01:37,656 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker > 2025-05-29 21:01:37,788 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29 21:01:37,816 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 3. Remaining time: 29993 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}]. > 2025-05-29 21:01:37,790 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29 21:01:37,913 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=16435239, minRoundTripTimeNanos=0} > 2025-05-29 21:01:38,047 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - Connected to MongoDB version 7.0.20 > 2025-05-29 21:01:39,076 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29 21:01:39,077 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1657403, minRoundTripTimeNanos=0} > 2025-05-29 21:01:39,077 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@b52b755, com.mongodb.Jep395RecordCodecProvider@a098d76, com.mongodb.KotlinCodecProvider@40e37b06]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29 21:01:39,078 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 12. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}]. > 2025-05-29 21:01:39,165 INFO : org.graylog2.plugin.system.FilePersistedNodeIdProvider - No node ID file found. Generated: ae83ff81-997b-49da-be53-a119b897458f > 2025-05-29 21:01:39,233 INFO : org.graylog2.configuration.IndexerDiscoveryProvider - No indexer hosts configured, using fallback http://127.0.0.1:9200 > 2025-05-29 21:01:39,529 INFO : org.graylog2.bootstrap.ServerBootstrap - Running 2 migrations... > 2025-05-29 21:01:39,549 INFO : org.graylog2.bootstrap.ServerBootstrap - Fresh installation detected, starting configuration webserver > 2025-05-29 21:01:39,555 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. > 2025-05-29 21:01:39,552 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 3 periodicals ... > 2025-05-29 21:01:39,554 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical] periodical in [2s], polling every [2s]. > 2025-05-29 21:01:39,555 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. > 2025-05-29 21:01:39,925 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null > 2025-05-29 21:01:40,125 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000] > 2025-05-29 21:01:40,126 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started. > 2025-05-29 21:01:40,129 INFO : org.graylog2.bootstrap.preflight.PreflightJerseyService - > --- > --- > --- > ######## ### ######### ########## #### #### --- .---- ---- > ############### ###################### ##### #### --- ------------ .----------- -- > ##### ###### ##### #### #### #### --- --- --- --- ----- > ############# #### ----- ---- > ###### #### ------- > #### #### #### ############ #### #### --- -- --- --- --- > > ### ### #### ############## #### #### --- --- -- -- -- > ======================================================================================================== > #### #### #### #### #### #### #### --- --- -- -- .-- > > ##### ##### #### #### #### ####### --- --- --- --- --- > It seems you are starting Graylog for the first time. To set up a fresh install, a setup interface has > ################ #### ############## ######- -- ---- ---- --- ----- > been started. You must log in to it to perform the initial configuration and continue. > ############## #### ############# ##### ----- ----------- ---------- -- > > Initial configuration is accessible at 0.0.0.0:9000, with username 'admin' and password 'redacted'. > #### #### --- > Try clicking on http://admin:redacted@0.0.0.0:9000 > ##### #### #### - .-- > > ======================================================================================================== > > ``` ---
**Author:** ![syntaxaire](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@syntaxaire](https://community.graylog.org/u/syntaxaire)\ **Post date:** [May 29, 2025, 9:11pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/4 "2025-05-29T21:11:55Z")
> **Datanode output** > > ```auto > Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/6f17995f-18d1-43db-8771-468e7554d232/vol_1h8ozzf79g74m0ul > Starting Container > 2025-05-29T21:13:21.627Z WARN [CmdLineTool] Couldn't create native lib dir . Unable to set ZstdTempFolder system property. > java.nio.file.AccessDeniedException: /native_libs > at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?] > at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] > at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] > at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?] > at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?] > at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?] > at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?] > at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?] > at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?] > at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?] > 2025-05-29T21:13:21.755Z INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {} > 2025-05-29T21:13:21.760Z WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins. > 2025-05-29T21:13:22.137Z INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true > 2025-05-29T21:13:22.138Z INFO [DatanodeBootstrap] Skipping preflight checks > at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:162) ~[graylog-datanode.jar:?] > at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:184) ~[graylog-datanode.jar:?] > ... 7 more > Caused by: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory) > at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?] > at java.base/java.io.FileInputStream.open(Unknown Source) ~[?:?] > at java.base/java.io.FileInputStream.(Unknown Source) ~[?:?] > at org.graylog.datanode.configuration.DatanodeKeystore.loadKeystore(DatanodeKeystore.java:157) ~[graylog-datanode.jar:?] > at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:184) ~[graylog-datanode.jar:?] > ... 7 more > 2025-05-29T21:13:23.369Z INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeConfigurationPeriodical] periodical in [0s], polling every [2s]. > 2025-05-29T21:13:23.370Z INFO [Periodicals] Starting [org.graylog.datanode.periodicals.OpensearchNodeHeartbeat] periodical in [0s], polling every [10s]. > 2025-05-29T21:13:23.370Z INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. > 2025-05-29T21:13:23.371Z INFO [Periodicals] Starting [org.graylog.datanode.periodicals.MetricsCollector] periodical in [0s], polling every [60s]. > 2025-05-29T21:13:23.414Z INFO [SearchableSnapshotsConfigurationBean] Opensearch snapshots not configured, skipping search role and cache configuration. > 2025-05-29T21:13:23.420Z INFO [TruststoreUtils] Detected existing JVM truststore: /usr/share/graylog-datanode/jvm/lib/security/cacerts of type pkcs12 > 2025-05-29T21:13:23.440Z INFO [OpensearchProcessService] > > ======================================================================================================== > It seems you are starting Data node for the first time. The current configuration is not sufficient to > start the indexer process because a security configuration is missing. You have to either provide http > and transport SSL certificates or use the Graylog preflight interface to configure this Data node remotely. > ======================================================================================================== > > 2025-05-29T21:13:23.441Z INFO [JerseyService] Starting Data node REST API > 2025-05-29T21:13:23.441Z INFO [DatanodeBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, OpensearchProcessService [RUNNING]=0, PeriodicalsService [RUNNING]=11, OpensearchConfigurationService [RUNNING]=76} > 2025-05-29T21:13:23.442Z INFO [DatanodeBootstrap] Graylog DataNode datanode up and running. > 2025-05-29T21:13:24.017Z INFO [Version] HV000001: Hibernate Validator 8.0.2.Final > 2025-05-29T21:13:24.293Z INFO [NetworkListener] Started listener bound to [[::]:8999] > 2025-05-29T21:13:22.716Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3850e90c, com.mongodb.Jep395RecordCodecProvider@3d9f5016, com.mongodb.KotlinCodecProvider@7e91ed74]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29T21:13:24.295Z INFO [HttpServer] [HttpServer] Started. > 2025-05-29T21:13:24.296Z INFO [JerseyService] Started REST API at <:::8999> > 2025-05-29T21:13:22.720Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3850e90c, com.mongodb.Jep395RecordCodecProvider@3d9f5016, com.mongodb.KotlinCodecProvider@7e91ed74]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} > 2025-05-29T21:13:22.751Z INFO [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29991 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}]. > 2025-05-29T21:13:22.789Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17555410, minRoundTripTimeNanos=0} > 2025-05-29T21:13:22.901Z INFO [DatanodeDirectories] Opensearch of the node e5325ad9-9355-40ef-b558-802c140a795c uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/lib/graylog-datanode/opensearch/logs', configurationSourceDir='Optional.empty', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'} > 2025-05-29T21:13:23.324Z INFO [DatanodeBootstrap] Graylog Data Node 6.2.2+13ba949 starting up (command: datanode) > 2025-05-29T21:13:23.325Z INFO [DatanodeBootstrap] JRE: Eclipse Adoptium 17.0.15 on Linux 6.12.12+bpo-cloud-amd64 > 2025-05-29T21:13:23.325Z INFO [DatanodeBootstrap] Deployment: container > 2025-05-29T21:13:23.325Z INFO [DatanodeBootstrap] OS: Ubuntu 22.04.5 LTS (jammy) > 2025-05-29T21:13:23.326Z INFO [DatanodeBootstrap] Arch: amd64 > 2025-05-29T21:13:23.362Z INFO [PeriodicalsService] Starting 6 periodicals ... > 2025-05-29T21:13:23.363Z INFO [PeriodicalsService] Delaying start of 0 periodicals until this node becomes leader ... > 2025-05-29T21:13:23.364Z INFO [Periodicals] Starting [org.graylog.datanode.bootstrap.preflight.DataNodeCertRenewalPeriodical] periodical in [0s], polling every [2s]. > 2025-05-29T21:13:23.368Z ERROR [PeriodicalsService] Could not initialize periodical. > java.lang.RuntimeException: org.graylog.datanode.configuration.DatanodeKeystoreException: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory) > at org.graylog.datanode.configuration.DatanodeKeystore.getCertificateExpiration(DatanodeKeystore.java:192) ~[graylog-datanode.jar:?] > at org.graylog.datanode.periodicals.NodePingPeriodical.registerServer(NodePingPeriodical.java:163) ~[graylog-datanode.jar:?] > at org.graylog.datanode.periodicals.NodePingPeriodical.initialize(NodePingPeriodical.java:134) ~[graylog-datanode.jar:?] > at org.graylog.datanode.initializers.PeriodicalsService.startPeriodicals(PeriodicalsService.java:76) ~[graylog-datanode.jar:?] > at org.graylog.datanode.initializers.PeriodicalsService.startUp(PeriodicalsService.java:61) ~[graylog-datanode.jar:?] > at com.google.common.util.concurrent.AbstractIdleService$DelegateService.lambda$doStart$0(AbstractIdleService.java:63) ~[guava-33.4.7-jre.jar:?] > at com.google.common.util.concurrent.Callables.lambda$threadRenaming$1(Callables.java:104) ~[guava-33.4.7-jre.jar:?] > at java.base/java.lang.Thread.run(Unknown Source) [?:?] > Caused by: org.graylog.datanode.configuration.DatanodeKeystoreException: java.io.FileNotFoundException: /var/lib/graylog-datanode/opensearch/config/keystore.jks (No such file or directory) > > ``` ---
**Author:** ![Tdvorak](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tdvorak](https://community.graylog.org/u/Tdvorak)\ **Post date:** [June 3, 2025, 12:10pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/5 "2025-06-03T12:10:45Z")
Hey @syntaxaire, By skipping the preflight checks you are also skipping a part of the code that initially creates the datanode keystore. Then datanode fails when trying to persist its certificate. Is there any reason why you are disabling preflight checks? I’d suggest enabling them again and start the preflight/provisioning. Then everything should be good. ---
**Author:** ![syntaxaire](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@syntaxaire](https://community.graylog.org/u/syntaxaire)\ **Post date:** [June 3, 2025, 1:42pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/6 "2025-06-03T13:42:37Z")
I have preflight checks disabled since is the only way I can run the datanode. Otherwise: ```auto Exception in thread "main" org.graylog2.bootstrap.preflight.PreflightCheckException: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl) at org.graylog.datanode.bootstrap.preflight.OpenSearchPreconditionsCheck.runCheck(OpenSearchPreconditionsCheck.java:51) at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421) at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52) at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:82) at org.graylog.datanode.bootstrap.DatanodeBootstrap.beforeInjectorCreation(DatanodeBootstrap.java:70) at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358) at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) at org.graylog.datanode.bootstrap.Main.main(Main.java:59) ``` It seems the solution is to set `node.store.allow_mmap: "false"` in `elasticsearch.yml`. Since the GRAYLOG\_DATANODE environment variables are not documented I’m not sure if there is a variable for this, or whether I need to create a custom datanode image to apply the change. (Edit: mmap is [not configurable](https://station.railway.com/questions/pro-plan-elastic-search-max-virtual-memo-b4067b4c) on Railway) ---
**Author:** ![Tdvorak](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tdvorak](https://community.graylog.org/u/Tdvorak)\ **Post date:** [June 3, 2025, 1:50pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/7 "2025-06-03T13:50:06Z")
Ah, I see, thanks! You should be able to configure this in [Data Node Configuration Overrides](https://go2docs.graylog.org/current/setting_up_graylog/data_node_configuration_overrides.htm) . Then you can probably re-enable the checks. ---
**Author:** ![syntaxaire](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@syntaxaire](https://community.graylog.org/u/syntaxaire)\ **Post date:** [June 4, 2025, 4:19pm UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/8 "2025-06-04T16:19:16Z")
I configured it like so but the preflight mmap check still fails. I set a custom start command to avoid making an entire new image: ```auto /bin/sh -c "cd /etc/graylog/datanode && echo \"node.store.allow_mmap=false\" >> overrideconfig.conf && chown graylog: overrideconfig.conf && echo \"\\nopensearch_configuration_overrides_file = /etc/graylog/datanode/overrideconfig.conf\" >> datanode.conf && ls -l && cat datanode.conf && cat overrideconfig.conf && cd && /entrypoint.sh graylog-datanode" ``` See the output of the `ls` and `cat` commands at the beginning of the deployment log: ```auto Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/74b021d7-bc76-4a5b-8e78-ed0a50063931/vol_1h8ozzf79g74m0ul total 16 -rw-r--r-- 1 graylog graylog 850 Jun 4 18:56 datanode.conf -rw-r--r-- 1 graylog graylog 444 May 7 11:18 jvm.options -rw-r--r-- 1 graylog graylog 1500 May 7 11:18 log4j2.xml -rw-r--r-- 1 graylog graylog 280 Jun 4 18:56 overrideconfig.conf opensearch_configuration_overrides_file = /etc/graylog/datanode/overrideconfig.conf node.store.allow_mmap=false 2025-06-04T16:09:16.122Z WARN [CmdLineTool] Couldn't create native lib dir . Unable to set ZstdTempFolder system property. java.nio.file.AccessDeniedException: /native_libs at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?] at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?] at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?] at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?] 2025-06-04T16:09:16.230Z INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {} 2025-06-04T16:09:16.234Z WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins. 2025-06-04T16:09:16.569Z INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true 2025-06-04T16:09:16.950Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} 2025-06-04T16:09:17.093Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17689048, minRoundTripTimeNanos=0} 2025-06-04T16:09:16.953Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} 2025-06-04T16:09:16.983Z INFO [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29991 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}]. 2025-06-04T16:09:17.189Z INFO [DatanodeDirectories] Opensearch of the node e5325ad9-9355-40ef-b558-802c140a795c uses following directories as its storage: DatanodeDirectories{dataTargetDir='/var/lib/graylog-datanode/opensearch/data', logsTargetDir='/var/lib/graylog-datanode/opensearch/logs', configurationSourceDir='Optional.empty', configurationTargetDir='/var/lib/graylog-datanode/opensearch/config'} 2025-06-04T16:09:22.370Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=mongodb-7c80a5be.railway.internal:27017, type=STANDALONE, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1862446, minRoundTripTimeNanos=0} 2025-06-04T16:09:17.235Z INFO [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.4.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "6.12.12+bpo-cloud-amd64"}, "platform": "Java/Eclipse Adoptium/17.0.15+6"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='mongo', source='graylog', password=, mechanismProperties=}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@3e906375, com.mongodb.Jep395RecordCodecProvider@40230eb9, com.mongodb.KotlinCodecProvider@5b7aa898]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-7c80a5be.railway.internal:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null} 2025-06-04T16:09:22.370Z INFO [cluster] Waiting for server to become available for operation with ID 11. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=mongodb-7c80a5be.railway.internal:27017, type=UNKNOWN, state=CONNECTING}]. 2025-06-04T16:09:22.411Z INFO [MongoDBPreflightCheck] Connected to MongoDB version 7.0.20 2025-06-04T16:09:22.421Z ERROR [PreflightCheckService] Preflight check failed with error: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl) 2025-06-04T16:09:22.421Z ERROR [CmdLineTool] Startup error: org.graylog2.bootstrap.preflight.PreflightCheckException: /proc/sys/vm/max_map_count value should be at least 262144 but is 65530 (set via "vm.max_map_count" sysctl) at org.graylog.datanode.bootstrap.preflight.OpenSearchPreconditionsCheck.runCheck(OpenSearchPreconditionsCheck.java:51) ~[graylog-datanode.jar:?] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:421) ~[guava-33.4.7-jre.jar:?] at org.graylog2.bootstrap.preflight.PreflightCheckService.runChecks(PreflightCheckService.java:52) ~[graylog2-server-6.2.2.jar:?] at org.graylog.datanode.bootstrap.DatanodeBootstrap.runPreFlightChecks(DatanodeBootstrap.java:82) ~[graylog-datanode.jar:?] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?] at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source) ~[?:?] at java.base/java.nio.file.Files.createDirectories(Unknown Source) ~[?:?] at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:306) [graylog2-server-6.2.2.jar:?] at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:283) [graylog2-server-6.2.2.jar:?] at org.graylog.datanode.bootstrap.Main.main(Main.java:59) [graylog-datanode.jar:?] 2025-06-04T16:09:21.415Z INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {} 2025-06-04T16:09:21.419Z WARN [PluginLoader] Plugin directory /plugin does not exist, not loading plugins. 2025-06-04T16:09:21.752Z INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms1g -Xmx1g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true ``` ---
**Author:** ![Tdvorak](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tdvorak](https://community.graylog.org/u/Tdvorak)\ **Post date:** [June 5, 2025, 8:53am UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/9 "2025-06-05T08:53:39Z")
Thanks, you are right. I just replicated the problem locally. The failing check is already on the datanode side. There is no way how to skip it without also skipping the keystore check as well. The only way how to fix that is a small change in the datanode code. I prepared something, will discuss the problem and solution with my colleagues and will let you know what happens next. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [June 19, 2025, 8:54am UTC](https://community.graylog.org/t/preflight-unconfigured-datanode-appears-once-in-list-then-disappears/35760/10 "2025-06-19T08:54:27Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.