Graylog doesn't work when 1 of my 2 mongodb instances is down:

Graylog Central (peer support)
1

1. Describe your incident:

When i stop 1 of my 2 mongodb instances, graylog doesn’t respond anymore.

2. Describe your environment:

2 graylog nodes (kubernetes containers)
3 mongodb nodes (2 nodes + 1 arbiter replica set) (kubernetes containers)
3 opensearch nodes (kubernetes containers)

  • OS Information:
    Fedora CoreOS 41.20250105.3.0
    Kubernetes v1.31.4

  • Package Version:
    Graylog 6.2.1
    Mongodb 7.0.12
    OpenSearch 2.15.0

  • Service logs, configurations, and environment variables:

mongodb_uri = mongodb://user:password@mongodb-0.mongodb.mongodb:27017,mongodb-1.mongodb.mongodb:27017/graylog?replicaSet=rs0&tls=true&tlsCAFile=/certs/ca.crt

Logs:

[pod/graylog-0/graylog] 2025-05-06 14:56:42,449 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoNodeIsRecoveringException: Command failed with error 91 (ShutdownInProgress): ‘The server is in quiesce mode and will shut down’ on server mongodb-1.mongodb.mongodb:27017. The full response is {“topologyVersion”: {“processId”: {“$oid”: “6819ff34cba3bd971044d31a”}, “counter”: 5}, “ok”: 0.0, “errmsg”: “The server is in quiesce mode and will shut down”, “code”: 91, “codeName”: “ShutdownInProgress”, “remainingQuiesceTimeMillis”: 15000, “$clusterTime”: {“clusterTime”: {“$timestamp”: {“t”: 1746536202, “i”: 25}}, “signature”: {“hash”: {“$binary”: {“base64”: “EovQAeAb8WuzGdVrZrClNCHEFGE=”, “subType”: “00”}}, “keyId”: 7500946085070766085}}, “operationTime”: {“$timestamp”: {“t”: 1746536202, “i”: 25}}}
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:264) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:206) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:520) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.receive(InternalStreamConnection.java:469) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:249) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] 2025-05-06 14:56:52,466 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:75) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:100) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] Caused by: java.net.UnknownHostException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultInetAddressResolver.lookupByName(DefaultInetAddressResolver.java:34) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:71) ~[graylog.jar:?]
[pod/graylog-0/graylog] … 5 more

3. What steps have you already taken to try and solve the problem?

If start the secondary mongodb instance, after that i see in the logs some api timeouts and the reconnectionn to the secondary mongodb instance and all goes back to normal.

[pod/graylog-0/graylog] 2025-05-06 15:01:30,874 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1032 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:30,896 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1101 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:30,922 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1070 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:30,929 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1098 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:32,624 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-1.mongodb.mongodb:27017, type=REPLICA_SET_SECONDARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=69607725, minRoundTripTimeNanos=0, setName=‘rs0’, canonicalAddress=mongodb-1.mongodb.mongodb:27017, hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], passives=, arbiters=[mongodb-arbiter.mongodb:27017], primary=‘mongodb-0.mongodb.mongodb:27017’, tagSet=TagSet{}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=681a081f092c3c91e22e5dac, counter=5}, lastWriteDate=Tue May 06 15:01:32 CEST 2025, lastUpdateTimeNanos=170936034447956}
[pod/graylog-0/graylog] 2025-05-06 15:01:34,251 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1004 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,277 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1029 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,306 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1005 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,306 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1027 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,320 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1018 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,321 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1043 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,394 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1010 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,404 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1020 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,527 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: Read timed out (duration: 1040 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,525 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1030 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,650 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1010 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:34,675 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1033 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:36,761 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1051 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:37,050 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1051 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:37,071 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1072 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:37,749 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1008 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:37,866 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1126 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:38,613 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1032 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:38,616 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1034 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:40,295 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1001 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:40,303 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1011 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:40,739 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1004 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:40,747 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1003 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,074 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: Read timed out (duration: 1040 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,077 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1039 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,601 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1053 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,603 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1056 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,610 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1007 ms)
[pod/graylog-0/graylog] 2025-05-06 15:01:41,650 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Failed to call API on node , cause: timeout (duration: 1047 ms)

4. How can the community help?

Any idea what might be the problem?

2

Logs when i start graylog with secondary mongodb instance down:

[pod/graylog-0/graylog] adding environment opts
15:24:17,564 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[show_security_events_in_pedt=off, data_tiering_cloud=off, preflight_web=on, configurable_value_units=on, setup_mode=on, cloud_inputs=on, investigation_report_by_ai=on, show_executive_dashboard_page=off, composable_index_templates=off, data_node_migration=on, remote_reindex_migration=off, instant_archiving=off, data_warehouse_search=on, new_report_creation=on, threat_coverage=on]}
[pod/graylog-0/graylog] 2025-05-06 15:24:17,951 WARN : org.mongodb.driver.uri - Connection string contains unsupported option ‘tlscafile’.
[pod/graylog-0/graylog] 2025-05-06 15:24:18,406 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 6.2.1+f599f1d [org.graylog.aws.AWSPlugin]
[pod/graylog-0/graylog] 2025-05-06 15:24:18,408 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Integrations 6.2.1+f599f1d [org.graylog.integrations.IntegrationsPlugin]
[pod/graylog-0/graylog] 2025-05-06 15:24:18,410 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 6.2.1+f599f1d [org.graylog.plugins.threatintel.ThreatIntelPlugin]
[pod/graylog-0/graylog] 2025-05-06 15:24:18,411 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 7 Support 6.2.1+f599f1d [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
[pod/graylog-0/graylog] 2025-05-06 15:24:18,412 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: OpenSearch 2 Support 6.2.1+f599f1d [org.graylog.storage.opensearch2.OpenSearch2Plugin]
[pod/graylog-0/graylog] 2025-05-06 15:24:18,427 WARN : org.mongodb.driver.uri - Connection string contains unsupported option ‘tlscafile’.
[pod/graylog-0/graylog] 2025-05-06 15:24:18,434 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
[pod/graylog-0/graylog] 2025-05-06 15:24:18,534 WARN : org.mongodb.driver.uri - Connection string contains unsupported option ‘tlscafile’.
[pod/graylog-0/graylog] 2025-05-06 15:24:18,571 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-0.mongodb.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:18,613 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-1.mongodb.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:18,678 INFO : org.mongodb.driver.client - MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.4.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.12.7-200.fc41.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.15+6”, “env”: {“container”: {“orchestrator”: “kubernetes”}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName=‘graylog’, source=‘graylog’, password=, mechanismProperties=}, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2cae9b8, com.mongodb.Jep395RecordCodecProvider@1457fde, com.mongodb.KotlinCodecProvider@6f94fb9d]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
[pod/graylog-0/graylog] 2025-05-06 15:24:18,683 INFO : org.mongodb.driver.client - MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.4.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.12.7-200.fc41.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.15+6”, “env”: {“container”: {“orchestrator”: “kubernetes”}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName=‘graylog’, source=‘graylog’, password=, mechanismProperties=}, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2cae9b8, com.mongodb.Jep395RecordCodecProvider@1457fde, com.mongodb.KotlinCodecProvider@6f94fb9d]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
[pod/graylog-0/graylog] 2025-05-06 15:24:18,735 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 4. Remaining time: 29987 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=mongodb-0.mongodb.mongodb:27017, type=UNKNOWN, state=CONNECTING}, {address=mongodb-1.mongodb.mongodb:27017, type=UNKNOWN, state=CONNECTING}].
[pod/graylog-0/graylog] 2025-05-06 15:24:18,831 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:75) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:100) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] Caused by: java.net.UnknownHostException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultInetAddressResolver.lookupByName(DefaultInetAddressResolver.java:34) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:71) ~[graylog.jar:?]
[pod/graylog-0/graylog] … 5 more
[pod/graylog-0/graylog] 2025-05-06 15:24:19,141 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-0.mongodb.mongodb:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=206744176, minRoundTripTimeNanos=0, setName=‘rs0’, canonicalAddress=mongodb-0.mongodb.mongodb:27017, hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], passives=, arbiters=[mongodb-arbiter.mongodb:27017], primary=‘mongodb-0.mongodb.mongodb:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000005, setVersion=2, topologyVersion=TopologyVersion{processId=6819f79f59e58e255525cc31, counter=6}, lastWriteDate=Tue May 06 15:24:18 CEST 2025, lastUpdateTimeNanos=172302524656524}
[pod/graylog-0/graylog] 2025-05-06 15:24:19,151 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-arbiter.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:19,161 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongodb-0.mongodb.mongodb:27017 with max election id 7fffffff0000000000000005 and max set version 2
[pod/graylog-0/graylog] 2025-05-06 15:24:19,223 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-arbiter.mongodb:27017, type=REPLICA_SET_ARBITER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=53538471, minRoundTripTimeNanos=0, setName=‘rs0’, canonicalAddress=mongodb-arbiter.mongodb:27017, hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], passives=, arbiters=[mongodb-arbiter.mongodb:27017], primary=‘mongodb-0.mongodb.mongodb:27017’, tagSet=TagSet{}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=6819f7a546892347efa48113, counter=2}, lastWriteDate=Tue May 06 15:14:35 CEST 2025, lastUpdateTimeNanos=172302631683777}
[pod/graylog-0/graylog] 2025-05-06 15:24:19,356 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:75) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:100) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] Caused by: java.net.UnknownHostException: mongodb-1.mongodb.mongodb
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultInetAddressResolver.lookupByName(DefaultInetAddressResolver.java:34) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:71) ~[graylog.jar:?]
[pod/graylog-0/graylog] … 5 more
[pod/graylog-0/graylog] 2025-05-06 15:24:19,424 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - Connected to MongoDB version 7.0.12
[pod/graylog-0/graylog] 2025-05-06 15:24:20,129 WARN : org.mongodb.driver.uri - Connection string contains unsupported option ‘tlscafile’.
[pod/graylog-0/graylog] 2025-05-06 15:24:20,132 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-0.mongodb.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:20,144 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-1.mongodb.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:20,151 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:75) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:100) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] Caused by: java.net.UnknownHostException: mongodb-1.mongodb.mongodb
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$CachedAddresses.get(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultInetAddressResolver.lookupByName(DefaultInetAddressResolver.java:34) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:71) ~[graylog.jar:?]
[pod/graylog-0/graylog] … 5 more
[pod/graylog-0/graylog] 2025-05-06 15:24:20,154 INFO : org.mongodb.driver.client - MongoClient with metadata {“driver”: {“name”: “mongo-java-driver|legacy”, “version”: “5.4.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.12.7-200.fc41.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.15+6”, “env”: {“container”: {“orchestrator”: “kubernetes”}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName=‘graylog’, source=‘graylog’, password=, mechanismProperties=}, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2cae9b8, com.mongodb.Jep395RecordCodecProvider@1457fde, com.mongodb.KotlinCodecProvider@6f94fb9d]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
[pod/graylog-0/graylog] 2025-05-06 15:24:20,162 INFO : org.mongodb.driver.client - MongoClient with metadata {“driver”: {“name”: “mongo-java-driver”, “version”: “5.4.0”}, “os”: {“type”: “Linux”, “name”: “Linux”, “architecture”: “amd64”, “version”: “6.12.7-200.fc41.x86_64”}, “platform”: “Java/Eclipse Adoptium/17.0.15+6”, “env”: {“container”: {“orchestrator”: “kubernetes”}}} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName=‘graylog’, source=‘graylog’, password=, mechanismProperties=}, transportSettings=null, commandListeners=, codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2cae9b8, com.mongodb.Jep395RecordCodecProvider@1457fde, com.mongodb.KotlinCodecProvider@6f94fb9d]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName=‘rs0’, serverSelector=‘null’, clusterListeners=‘’, serverSelectionTimeout=‘30000 ms’, localThreshold=‘15 ms’}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=, maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverMonitoringMode=AUTO, serverListeners=‘’, serverMonitorListeners=‘’}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=false, context=null}, applicationName=‘null’, compressorList=, uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null, timeoutMS=null}
[pod/graylog-0/graylog] 2025-05-06 15:24:20,165 INFO : org.mongodb.driver.cluster - Waiting for server to become available for operation with ID 17. Remaining time: 29999 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=mongodb-0.mongodb.mongodb:27017, type=UNKNOWN, state=CONNECTING}, {address=mongodb-1.mongodb.mongodb:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb}, caused by {java.net.UnknownHostException: mongodb-1.mongodb.mongodb}}].
[pod/graylog-0/graylog] 2025-05-06 15:24:20,169 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-0.mongodb.mongodb:27017, type=REPLICA_SET_PRIMARY, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=22361892, minRoundTripTimeNanos=0, setName=‘rs0’, canonicalAddress=mongodb-0.mongodb.mongodb:27017, hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], passives=, arbiters=[mongodb-arbiter.mongodb:27017], primary=‘mongodb-0.mongodb.mongodb:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000005, setVersion=2, topologyVersion=TopologyVersion{processId=6819f79f59e58e255525cc31, counter=6}, lastWriteDate=Tue May 06 15:24:18 CEST 2025, lastUpdateTimeNanos=172303579940193}
[pod/graylog-0/graylog] 2025-05-06 15:24:20,171 INFO : org.mongodb.driver.cluster - Adding discovered server mongodb-arbiter.mongodb:27017 to client view of cluster
[pod/graylog-0/graylog] 2025-05-06 15:24:20,175 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongodb-0.mongodb.mongodb:27017 with max election id 7fffffff0000000000000005 and max set version 2
[pod/graylog-0/graylog] 2025-05-06 15:24:20,222 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongodb-arbiter.mongodb:27017, type=REPLICA_SET_ARBITER, cryptd=false, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=21, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=40470305, minRoundTripTimeNanos=0, setName=‘rs0’, canonicalAddress=mongodb-arbiter.mongodb:27017, hosts=[mongodb-0.mongodb.mongodb:27017, mongodb-1.mongodb.mongodb:27017], passives=, arbiters=[mongodb-arbiter.mongodb:27017], primary=‘mongodb-0.mongodb.mongodb:27017’, tagSet=TagSet{}, electionId=null, setVersion=2, topologyVersion=TopologyVersion{processId=6819f7a546892347efa48113, counter=2}, lastWriteDate=Tue May 06 15:14:35 CEST 2025, lastUpdateTimeNanos=172303630574484}
[pod/graylog-0/graylog] 2025-05-06 15:24:30,662 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongodb-1.mongodb.mongodb:27017
[pod/graylog-0/graylog] com.mongodb.MongoSocketException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:75) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:100) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:79) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:233) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.lookupServerDescription(DefaultServerMonitor.java:219) [graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitor.run(DefaultServerMonitor.java:176) [graylog.jar:?]
[pod/graylog-0/graylog] Caused by: java.net.UnknownHostException: mongodb-1.mongodb.mongodb: Name or service not known
[pod/graylog-0/graylog] at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at java.base/java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.DefaultInetAddressResolver.lookupByName(DefaultInetAddressResolver.java:34) ~[graylog.jar:?]
[pod/graylog-0/graylog] at com.mongodb.internal.connection.ServerAddressHelper.getSocketAddresses(ServerAddressHelper.java:71) ~[graylog.jar:?]
[pod/graylog-0/graylog] … 5 more

3

Also i can connect to mongodb with mongosh without any issues when the secondary instance is down:

mongosh graylog --host mongodb.k8s.azar.pt -u user -p password --tls --eval “db.nodes.find()”
[
{
_id: ObjectId(‘681a082959e58e25552f8a1c’),
node_id: ‘f00cf80d-3ec3-4ea6-aac6-fa79d9e586f4’,
hostname: ‘graylog-0.graylog.graylog.svc.cluster.local’,
is_leader: true,
last_seen: Timestamp({ t: 1746537275, i: 11 }),
transport_address: ‘http://graylog-0.graylog.graylog:9000/api/’
},
{
_id: ObjectId(‘681a0ae359e58e2555322534’),
node_id: ‘f00cf80e-3ec3-4ea6-aac6-fa79d9e586f5’,
hostname: ‘graylog-1.graylog.graylog.svc.cluster.local’,
is_leader: false,
last_seen: Timestamp({ t: 1746537275, i: 6 }),
transport_address: ‘http://graylog-1.graylog.graylog:9000/api/’
}
]

mongosh admin --host mongodb.k8s.azar.pt -u user -p password --tls
Current Mongosh Log ID: 681a10252d4c82e9b41b26ff
Connecting to: mongodb://@mongodb.k8s.azar.pt:27017/admin?directConnection=true&tls=true&appName=mongosh+2.5.0
Using MongoDB: 7.0.12
Using Mongosh: 2.5.0

For mongosh info see: Welcome to MongoDB Shell (mongosh) - mongosh - MongoDB Docs

rs0 [direct: primary] admin> rs.status()
{
set: ‘rs0’,
date: ISODate(‘2025-05-06T13:35:35.521Z’),
myState: 1,
term: Long(‘5’),
syncSourceHost: ‘’,
syncSourceId: -1,
heartbeatIntervalMillis: Long(‘2000’),
majorityVoteCount: 2,
writeMajorityCount: 2,
votingMembersCount: 3,
writableVotingMembersCount: 2,
optimes: {
lastCommittedOpTime: { ts: Timestamp({ t: 1746537275, i: 5 }), t: Long(‘5’) },
lastCommittedWallTime: ISODate(‘2025-05-06T13:14:35.046Z’),
readConcernMajorityOpTime: { ts: Timestamp({ t: 1746537275, i: 5 }), t: Long(‘5’) },
appliedOpTime: { ts: Timestamp({ t: 1746538528, i: 1 }), t: Long(‘5’) },
durableOpTime: { ts: Timestamp({ t: 1746538528, i: 1 }), t: Long(‘5’) },
lastAppliedWallTime: ISODate(‘2025-05-06T13:35:28.912Z’),
lastDurableWallTime: ISODate(‘2025-05-06T13:35:28.912Z’)
},
lastStableRecoveryTimestamp: Timestamp({ t: 1746537275, i: 5 }),
electionCandidateMetrics: {
lastElectionReason: ‘electionTimeout’,
lastElectionDate: ISODate(‘2025-05-06T11:51:08.640Z’),
electionTerm: Long(‘5’),
lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 0, i: 0 }), t: Long(‘-1’) },
lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1746532081, i: 1 }), t: Long(‘4’) },
numVotesNeeded: 2,
priorityAtElection: 1,
electionTimeoutMillis: Long(‘10000’),
numCatchUpOps: Long(‘0’),
newTermStartDate: ISODate(‘2025-05-06T11:51:08.681Z’),
wMajorityWriteAvailabilityDate: ISODate(‘2025-05-06T11:51:30.152Z’)
},
members: [
{
_id: 0,
name: ‘mongodb-0.mongodb.mongodb:27017’,
health: 1,
state: 1,
stateStr: ‘PRIMARY’,
uptime: 6280,
optime: { ts: Timestamp({ t: 1746538528, i: 1 }), t: Long(‘5’) },
optimeDate: ISODate(‘2025-05-06T13:35:28.000Z’),
lastAppliedWallTime: ISODate(‘2025-05-06T13:35:28.912Z’),
lastDurableWallTime: ISODate(‘2025-05-06T13:35:28.912Z’),
syncSourceHost: ‘’,
syncSourceId: -1,
infoMessage: ‘’,
electionTime: Timestamp({ t: 1746532268, i: 1 }),
electionDate: ISODate(‘2025-05-06T11:51:08.000Z’),
configVersion: 2,
configTerm: 5,
self: true,
lastHeartbeatMessage: ‘’
},
{
_id: 1,
name: ‘mongodb-1.mongodb.mongodb:27017’,
health: 0,
state: 8,
stateStr: ‘(not reachable/healthy)’,
uptime: 0,
optime: { ts: Timestamp({ t: 0, i: 0 }), t: Long(‘-1’) },
optimeDurable: { ts: Timestamp({ t: 0, i: 0 }), t: Long(‘-1’) },
optimeDate: ISODate(‘1970-01-01T00:00:00.000Z’),
optimeDurableDate: ISODate(‘1970-01-01T00:00:00.000Z’),
lastAppliedWallTime: ISODate(‘2025-05-06T13:14:35.046Z’),
lastDurableWallTime: ISODate(‘2025-05-06T13:14:35.046Z’),
lastHeartbeat: ISODate(‘2025-05-06T13:35:34.361Z’),
lastHeartbeatRecv: ISODate(‘2025-05-06T13:14:35.060Z’),
pingMs: Long(‘0’),
lastHeartbeatMessage: ‘Error connecting to mongodb-1.mongodb.mongodb:27017 :: caused by :: Could not find address for mongodb-1.mongodb.mongodb:27017: SocketException: onInvoke :: caused by :: Host not found (authoritative)’,
syncSourceHost: ‘’,
syncSourceId: -1,
infoMessage: ‘’,
configVersion: 2,
configTerm: 5
},
{
_id: 2,
name: ‘mongodb-arbiter.mongodb:27017’,
health: 1,
state: 7,
stateStr: ‘ARBITER’,
uptime: 6267,
lastHeartbeat: ISODate(‘2025-05-06T13:35:34.698Z’),
lastHeartbeatRecv: ISODate(‘2025-05-06T13:35:35.329Z’),
pingMs: Long(‘0’),
lastHeartbeatMessage: ‘’,
syncSourceHost: ‘’,
syncSourceId: -1,
infoMessage: ‘’,
configVersion: 2,
configTerm: 5
}
],
ok: 1,
‘$clusterTime’: {
clusterTime: Timestamp({ t: 1746538528, i: 1 }),
signature: {
hash: Binary.createFromBase64(‘oLkSL/PzAej2zlF0I2tEz9jwAq8=’, 0),
keyId: Long(‘7500946085070766085’)
}
},
operationTime: Timestamp({ t: 1746538528, i: 1 })
}
rs0 [direct: primary] admin>

4

Seems the problem is on mongodb not graylog, though i can connect to mongodb when secondary node is down, it hangs when trying to insert any record.

I will try to figure out what is wrong with mongodb setup.

5

Problem solved, i had to change the mongodb defaultWriteConcern from ‘majority’ to 1

db.adminCommand({ “setDefaultRWConcern” : 1, “defaultWriteConcern” : { “w” : 1 } })

2 Likes