Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
Error is happening in preflight.
The error appears:
Unable to retrieve version from indexer node: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:9200. - Connection refused.;Unable to retrieve version from indexer node: : Retrying failed to complete successfully after 1 attempts.
2. Describe your environment:
-
OS Information:
Oracle Linux
Installation in2 servers
1-Mongo +graylog server
2-Datanode -
Package Version:
Graylog 6.1
Mongo 2.4.0 -
Service logs, configurations, and environment variables:
Config Server:
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = password
root_password_sha2 = password
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 172.20.128.94:9000
stream_aware_field_types=false
disabled_retention_strategies = none,close
allow_leading_wildcard_searches = false
allow_highlighting = false
field_value_suggestion_mode = on
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 72h
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
integrations_scripts_dir = /usr/share/graylog-server/scripts
Config Mongo:
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
storage:
dbPath: /var/lib/mongo
processManagement:
timeZoneInfo: /usr/share/zoneinfo
net:
port: 27017
bindIpAll: true
Config Datanode
node_id_file = /etc/graylog/datanode/node-id
config_location = /etc/graylog/datanode
password_secret = password
root_password_sha2 =
mongodb_uri = mongodb://172.20.128.94:27017/graylog
opensearch_http_port = 9200
opensearch_location = /usr/share/graylog-datanode/dist
opensearch_config_location = /var/lib/graylog-datanode/opensearch/config
opensearch_data_location = /var/lib/graylog-datanode/opensearch/data
opensearch_logs_location = /var/log/graylog-datanode/opensearch
opensearch_heap = 4g
LOGS
Server
2025-03-05T09:41:08.278-03:00 ERROR [DatanodeConnectivityCheck] Unable to retrieve version from indexer node: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:9200. - Connection refused.
2025-03-05T09:41:08.279-03:00 INFO [DatanodeConnectivityCheck] Indexer is not available. Retry #1/1
2025-03-05T09:41:08.279-03:00 ERROR [DatanodeConnectivityCheck] Unable to retrieve version from indexer node:
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 1 attempts.
at com.github.rholder.retry.Retryer.call(Retryer.java:174) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.probe(VersionProbe.java:107) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.DatanodeConnectivityCheck.probe(DatanodeConnectivityCheck.java:52) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.verifyActualConnection(PreflightResource.java:117) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.getProvisioningState(PreflightResource.java:108) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.lambda$listDataNodes$0(PreflightResource.java:95) ~[graylog.jar:?]
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog2.bootstrap.preflight.web.resources.PreflightResource.listDataNodes(PreflightResource.java:103) ~[graylog.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [graylog.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
----------Log Datanode
2025-03-05T09:45:09.813-03:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2025-03-05T09:45:10.093-03:00 INFO [CmdLineTool] Running with JVM arguments: -Dlog4j.configurationFile=file:///etc/graylog/datanode/log4j2.xml -Xms2g -Xmx2g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -Djdk.tls.acknowledgeCloseNotify=true
2025-03-05T09:45:10.221-03:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “5.15.0-305.176.4.el8uek.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@79316f3a, com.mongodb.Jep395RecordCodecProvider@381d7219, com.mongodb.KotlinCodecProvider@49fe3142]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[172.20.128.94:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-03-05T09:45:10.222-03:00 INFO [client] MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.2.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “5.15.0-305.176.4.el8uek.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.14+7”} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@79316f3a, com.mongodb.Jep395RecordCodecProvider@381d7219, com.mongodb.KotlinCodecProvider@49fe3142]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[172.20.128.94:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName=‘null’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
2025-03-05T09:45:10.245-03:00 INFO [cluster] Waiting for server to become available for operation with ID 3. Remaining time: 29995 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=UNKNOWN, servers=[{address=172.20.128.94:27017, type=UNKNOWN, state=CONNECTING}].
2025-03-05T09:45:20.237-03:00 INFO [cluster] Exception in monitor thread while connecting to server 172.20.128.94:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.lambda$open$0(SocketStream.java:85) ~[mongodb-driver-core-5.2.0.jar:?]
at java.base/java.util.Optional.orElseThrow(Unknown Source) ~[?:?]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:85) ~[mongodb-driver-core-5.2.0.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[mongodb-driver-core-5.2.0.jar:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [mongodb-driver-core-5.2.0.jar:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [mongodb-driver-core-5.2.0.jar:?]
Caused by: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source) ~[?:?]
at java.base/java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
at java.base/java.net.Socket.connect(Unknown Source) ~[?:?]
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:76) ~[mongodb-driver-core-5.2.0.jar:?]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:104) ~[mongodb-driver-core-5.2.0.jar:?]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[mongodb-driver-core-5.2.0.jar:?]
… 3 more
inux-x64]