Can't connect to web interface after upgrade to 6.0

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident: Upgraded Graylog from 5.0 to 6.0. Everything seems to be running properly except that the web service on Port 9000 is not acceptble (ERR_CONNECTION_REFUSED)

2. Describe your environment:

  • OS Information: RHEL 8

  • Package Version: Graylog 6, Opensearch 1.3.15, Mongo version 5.0.31

  • from graylog.conf

web_enable= true
is_master = true
node_id_file = /etc/graylog/server/node-id
root_password_sha2 = <>
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.x.x.x:9000
http_enable_tls = true
http_tls_cert_file = /etc/graylog/ssl/graylog01.pem
http_tls_key_file = /etc/graylog/ssl/graylog01-key.pem
elasticsearch_hosts = http://server.domain.address01:9200,http://server.domain.address02:9200, http://server.domain.address03:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 16gb
lb_recognition_period_seconds = 3
stale_master_timeout = 10000
mongodb_uri = mongodb://server.domain.address01/graylog
mongodb_uri = mongodb://graylogAdmin:.x.x.2:27017server.domain.address02:27017,server.domain.address03:27017/graylog?replicaSet=rs01
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
prometheus_exporter_enabled = true
transport_email_enabled = true
transport_email_hostname = 10.x.x.x
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_from_email = graylog@domain.com

m

3. What steps have you already taken to try and solve the problem?

lsof -i :9000 shows nothing. grep : 9000 /var/log/graylog/server/server.log shows nothing of importance (only an error from last week that occurred while I was updating certs on all three graylog servers. It’s like port 9000 just…went away after the upgrade I double-checked all of the matricies I could find and I couldn’t see any compatibility issues between graylog,opensearch, or mongo for my upgrade path.

4. How can the community help?

Help me figure out why nothing is listening on port 9000. Bonus points if you know exactly how to fix it.

2

Hey @dannymccaslin,

It sounds as though Graylog is failing the preflight check, post the server log at the point of a restart of the Graylog service.

I’ve seen this specifically occur with updates to 6.0 when using a Mongo replica set and one of the members of the replica set is unavailable.

3

Thanks for the reply. I did notice some SELinux findings for mongod, which was a little disturbing because this cluster has been running for years without issue. Anyway, resolved those. Here is the obfuscated output of the graylog server log from the latest service restart

2025-12-02T05:26:37.907-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {default properties file=[frontend_hotkeys=on, field_types_management=on, cloud_inputs=on, data_tiering_cloud=off, scripting_api_preview=on, composable_index_templates=off, search_filter=on, preflight_web=on, data_node_migration=off, instant_archiving=off]}
2025-12-02T05:26:38.510-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 6.0.14+508aa86 [org.graylog.aws.AWSPlugin]
2025-12-02T05:26:38.511-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 6.0.14+508aa86 [org.graylog.integrations.IntegrationsPlugin]
2025-12-02T05:26:38.511-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 6.0.14+508aa86 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2025-12-02T05:26:38.512-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 6.0.14+508aa86 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2025-12-02T05:26:38.512-05:00 INFO  [CmdLineTool] Loaded plugin: OpenSearch 2 Support 6.0.14+508aa86 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2025-12-02T05:26:38.534-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms10g -Xmx10g -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=rpm
2025-12-02T05:26:38.655-05:00 INFO  [cluster] Adding discovered server 10.x.x.2:27017 to client view of cluster
2025-12-02T05:26:38.695-05:00 INFO  [cluster] Adding discovered server node2.domain.com:27017 to client view of cluster
2025-12-02T05:26:38.701-05:00 INFO  [cluster] Adding discovered server node3.domain.com:27017 to client view of cluster
2025-12-02T05:26:38.735-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.0.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.87.1.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylogAdmin', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2aff9dff, com.mongodb.Jep395RecordCodecProvider@5f0f9947, com.mongodb.KotlinCodecProvider@1aad0b1]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[10.x.x.2:27017, node2.domain.com:27017, node3.domain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs01', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2025-12-02T05:26:38.739-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.0.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.87.1.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylogAdmin', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2aff9dff, com.mongodb.Jep395RecordCodecProvider@5f0f9947, com.mongodb.KotlinCodecProvider@1aad0b1]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[10.x.x.2:27017, node2.domain.com:27017, node3.domain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs01', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2025-12-02T05:26:38.772-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node3.domain.com:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=26889279, setName='rs01', canonicalAddress=node3.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebe0d47b859cdb11b6898, counter=4}, lastWriteDate=Tue Dec 02 05:26:30 EST 2025, lastUpdateTimeNanos=387314684242}
2025-12-02T05:26:38.772-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node2.domain.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=28371888, setName='rs01', canonicalAddress=node2.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000001b3, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebe0090a79dcb14d8f104, counter=6}, lastWriteDate=Tue Dec 02 05:26:30 EST 2025, lastUpdateTimeNanos=387318185113}
2025-12-02T05:26:38.774-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=10.x.x.2:27017, type=REPLICA_SET_OTHER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=30285677, setName='rs01', canonicalAddress=node1.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebd838cd4d9d58c58dc71, counter=5}, lastWriteDate=Fri Apr 04 22:11:52 EDT 2025, lastUpdateTimeNanos=387318633491}
2025-12-02T05:26:38.774-05:00 INFO  [cluster] Waiting for server to become available for operation with ID 7. Remaining time: 30000 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=10.x.x.2:27017, type=UNKNOWN, state=CONNECTING}, {address=node2.domain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=node3.domain.com:27017, type=UNKNOWN, state=CONNECTING}].
2025-12-02T05:26:38.777-05:00 INFO  [cluster] Adding discovered server node1.domain.com:27017 to client view of cluster
2025-12-02T05:26:38.782-05:00 INFO  [cluster] Server 10.x.x.2:27017 is no longer a member of the replica set.  Removing from client view of cluster.
2025-12-02T05:26:38.785-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node1.domain.com:27017, type=REPLICA_SET_OTHER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4586446, setName='rs01', canonicalAddress=node1.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebd838cd4d9d58c58dc71, counter=5}, lastWriteDate=Fri Apr 04 22:11:52 EDT 2025, lastUpdateTimeNanos=387335669438}
2025-12-02T05:26:38.788-05:00 INFO  [cluster] Discovered replica set primary node2.domain.com:27017 with max election id 7fffffff00000000000001b3 and max set version 38055
2025-12-02T05:26:38.956-05:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 5.0.31
2025-12-02T05:26:39.501-05:00 INFO  [cluster] Adding discovered server 10.x.x.2:27017 to client view of cluster
2025-12-02T05:26:39.506-05:00 INFO  [cluster] Adding discovered server node2.domain.com:27017 to client view of cluster
2025-12-02T05:26:39.510-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=10.x.x.2:27017, type=REPLICA_SET_OTHER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4449911, setName='rs01', canonicalAddress=node1.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebd838cd4d9d58c58dc71, counter=5}, lastWriteDate=Fri Apr 04 22:11:52 EDT 2025, lastUpdateTimeNanos=388060161468}
2025-12-02T05:26:39.515-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node2.domain.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2606961, setName='rs01', canonicalAddress=node2.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=7fffffff00000000000001b3, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebe0090a79dcb14d8f104, counter=6}, lastWriteDate=Tue Dec 02 05:26:30 EST 2025, lastUpdateTimeNanos=388064874494}
2025-12-02T05:26:39.517-05:00 INFO  [cluster] Adding discovered server node3.domain.com:27017 to client view of cluster
2025-12-02T05:26:39.527-05:00 INFO  [cluster] Adding discovered server node1.domain.com:27017 to client view of cluster
2025-12-02T05:26:39.527-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "5.0.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.87.1.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylogAdmin', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2aff9dff, com.mongodb.Jep395RecordCodecProvider@5f0f9947, com.mongodb.KotlinCodecProvider@1aad0b1]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[10.x.x.2:27017, node2.domain.com:27017, node3.domain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs01', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2025-12-02T05:26:39.530-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node3.domain.com:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3929807, setName='rs01', canonicalAddress=node3.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebe0d47b859cdb11b6898, counter=4}, lastWriteDate=Tue Dec 02 05:26:30 EST 2025, lastUpdateTimeNanos=388080210368}
2025-12-02T05:26:39.533-05:00 INFO  [cluster] Canonical address node1.domain.com:27017 does not match server address.  Removing 10.x.x.2:27017 from client view of cluster
2025-12-02T05:26:39.535-05:00 INFO  [client] MongoClient with metadata {"driver": {"name": "mongo-java-driver", "version": "5.0.0"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "4.18.0-553.87.1.el8_10.x86_64"}, "platform": "Java/Eclipse Adoptium/17.0.14+7"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='graylogAdmin', source='graylog', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@2aff9dff, com.mongodb.Jep395RecordCodecProvider@5f0f9947, com.mongodb.KotlinCodecProvider@1aad0b1]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[10.x.x.2:27017, node2.domain.com:27017, node3.domain.com:27017], srvServiceName=mongodb, mode=MULTIPLE, requiredClusterType=REPLICA_SET, requiredReplicaSetName='rs01', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2025-12-02T05:26:39.536-05:00 INFO  [cluster] Waiting for server to become available for operation with ID 24. Remaining time: 30000 ms. Selector: ReadPreferenceServerSelector{readPreference=primary}, topology description: {type=REPLICA_SET, servers=[{address=10.x.x.2:27017, type=UNKNOWN, state=CONNECTING}, {address=node2.domain.com:27017, type=UNKNOWN, state=CONNECTING}, {address=node3.domain.com:27017, type=UNKNOWN, state=CONNECTING}].
2025-12-02T05:26:39.537-05:00 INFO  [cluster] Discovered replica set primary node2.domain.com:27017 with max election id 7fffffff00000000000001b3 and max set version 38055
2025-12-02T05:26:39.538-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=node1.domain.com:27017, type=REPLICA_SET_OTHER, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3852241, setName='rs01', canonicalAddress=node1.domain.com:27017, hosts=[node3.domain.com:27017, node1.domain.com:27017, node2.domain.com:27017], passives=[], arbiters=[], primary='node2.domain.com:27017', tagSet=TagSet{[]}, electionId=null, setVersion=38055, topologyVersion=TopologyVersion{processId=692ebd838cd4d9d58c58dc71, counter=5}, lastWriteDate=Fri Apr 04 22:11:52 EDT 2025, lastUpdateTimeNanos=388088187695}
2025-12-02T05:26:40.351-05:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:1.3.15>
2025-12-02T05:26:40.567-05:00 INFO  [Version] HV000001: Hibernate Validator null
2025-12-02T05:26:42.996-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2025-12-02T05:26:43.180-05:00 INFO  [LogManager] Loading logs.
2025-12-02T05:26:43.205-05:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000061124590552.index, deleting and rebuilding index...
2025-12-02T05:26:43.395-05:00 INFO  [LogManager] Logs loading complete.
2025-12-02T05:26:43.398-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2025-12-02T05:26:43.433-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.

I know the mongo version says 5.0.0 there, but in mongosh I have:

Current Mongosh Log ID: 692ec28c667b7871b58de665
Connecting to:          mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.5.10
Using MongoDB:          5.0.31
Using Mongosh:          2.5.10
4

The compatibility matrix found here states that the minimum MongoDB version is 5.0.7 for Graylog 6.0. You seems to be on 5.0.31 although I’m not sure that is the root of the problem.

Could you run an rs.status() from the mongosh context?

5

Here you go:

{
  set: 'rs01',
  date: ISODate('2025-12-02T11:30:41.637Z'),
  myState: 3,
  term: Long('435'),
  tooStale: true,
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long('2000'),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 0, i: 0 }), t: Long('-1') },
    lastCommittedWallTime: ISODate('1970-01-01T00:00:00.000Z'),
    appliedOpTime: { ts: Timestamp({ t: 1743819112, i: 64 }), t: Long('398') },
    durableOpTime: { ts: Timestamp({ t: 1743819112, i: 64 }), t: Long('398') },
    lastAppliedWallTime: ISODate('2025-04-05T02:11:52.814Z'),
    lastDurableWallTime: ISODate('2025-04-05T02:11:52.814Z')
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1743819108, i: 13 }),
  electionParticipantMetrics: {
    votedForCandidate: true,
    electionTerm: Long('435'),
    lastVoteDate: ISODate('2025-12-02T10:23:19.997Z'),
    electionCandidateMemberId: 1,
    voteReason: '',
    lastAppliedOpTimeAtElection: { ts: Timestamp({ t: 1743819112, i: 64 }), t: Long('398') },
    maxAppliedOpTimeInSet: { ts: Timestamp({ t: 1764670910, i: 1 }), t: Long('434') },
    priorityAtElection: 1
  },
  members: [
    {
      _id: 0,
      name: 'node1.domain.com:27017',
      health: 1,
      state: 3,
      stateStr: 'RECOVERING',
      uptime: 4191,
      optime: { ts: Timestamp({ t: 1743819112, i: 64 }), t: Long('398') },
      optimeDate: ISODate('2025-04-05T02:11:52.000Z'),
      lastAppliedWallTime: ISODate('2025-04-05T02:11:52.814Z'),
      lastDurableWallTime: ISODate('2025-04-05T02:11:52.814Z'),
      syncSourceHost: '',
      syncSourceId: -1,
      maintenanceMode: 1,
      infoMessage: '',
      configVersion: 38055,
      configTerm: 435,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'node2.domain.com:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 4051,
      optime: { ts: Timestamp({ t: 1764675040, i: 1 }), t: Long('435') },
      optimeDurable: { ts: Timestamp({ t: 1764675040, i: 1 }), t: Long('435') },
      optimeDate: ISODate('2025-12-02T11:30:40.000Z'),
      optimeDurableDate: ISODate('2025-12-02T11:30:40.000Z'),
      lastAppliedWallTime: ISODate('2025-12-02T11:30:40.702Z'),
      lastDurableWallTime: ISODate('2025-12-02T11:30:40.702Z'),
      lastHeartbeat: ISODate('2025-12-02T11:30:41.622Z'),
      lastHeartbeatRecv: ISODate('2025-12-02T11:30:41.048Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1764671000, i: 1 }),
      electionDate: ISODate('2025-12-02T10:23:20.000Z'),
      configVersion: 38055,
      configTerm: 435    {
      _id: 1,
      name: 'node2.domain.com:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 4511,
      optime: { ts: Timestamp({ t: 1764675500, i: 1 }), t: Long('435') },
      optimeDurable: { ts: Timestamp({ t: 1764675500, i: 1 }), t: Long('435') },
      optimeDate: ISODate('2025-12-02T11:38:20.000Z'),
      optimeDurableDate: ISODate('2025-12-02T11:38:20.000Z'),
      lastAppliedWallTime: ISODate('2025-12-02T11:38:20.714Z'),
      lastDurableWallTime: ISODate('2025-12-02T11:38:20.714Z'),
      lastHeartbeat: ISODate('2025-12-02T11:38:21.310Z'),
      lastHeartbeatRecv: ISODate('2025-12-02T11:38:21.131Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1764671000, i: 1 }),
      electionDate: ISODate('2025-12-02T10:23:20.000Z'),
      configVersion: 38055,
      configTerm: 435
    },
    {
      _id: 2,
      name: 'node3.domain.com:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 4497,
      optime: { ts: Timestamp({ t: 1764675500, i: 1 }), t: Long('435') },
      optimeDurable: { ts: Timestamp({ t: 1764675500, i: 1 }), t: Long('435') },
      optimeDate: ISODate('2025-12-02T11:38:20.000Z'),
      optimeDurableDate: ISODate('2025-12-02T11:38:20.000Z'),
      lastAppliedWallTime: ISODate('2025-12-02T11:38:20.714Z'),
      lastDurableWallTime: ISODate('2025-12-02T11:38:20.714Z'),
      lastHeartbeat: ISODate('2025-12-02T11:38:21.310Z'),
      lastHeartbeatRecv: ISODate('2025-12-02T11:38:21.132Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: 'node2.domain.com:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 38055,
      configTerm: 435
    }
  ],
  ok: 1
}

    },

Sorry the formatting is weird. I still cant get the hang of copying from TMUX

6

No worries, formatting is fit for purpose. As you can see node 1 is in a recovering state, you should stop the mongodb service on node 1 and clear it’s data directory. Restart the service and it should sync from the primary and have an up to date DB.

Run the rs.status() again and check node1.domain.com now has state SECONDARY, if it does bounce the graylog service and see if it passes pre flight.

7

When you talk about clearing the data directory, should I clear out everything or just the .wt files? There are other files in here that I don’t know if I can remove, like the keyfile, as well as the journal and diagnostic directory.

]$ ls -la /var/lib/mongo | grep -v .wt
total 503928
drwxr-xr-x.  4 mongod mongod     12288 Dec  2 07:35 .
drwxr-xr-x. 57 root   root        4096 Jun  6  2024 ..
drwx------.  2 mongod mongod      4096 Dec  2 07:35 diagnostic.data
drwx------.  2 mongod mongod       110 Dec  2 05:20 journal
-rw-------.  1 mongod mongod      1004 Jun  6  2024 mongodb-keyfile
-rw-------.  1 mongod mongod         0 Dec  2 07:35 mongod.lock
-rw-------.  1 mongod mongod       114 Jun  6  2024 storage.bson
-rw-------.  1 mongod mongod        50 Jun  6  2024 WiredTiger
-rw-------.  1 mongod mongod        21 Jun  6  2024 WiredTiger.lock
-rw-------.  1 mongod mongod      1499 Dec  2 07:35 WiredTiger.turtle
8

I’ve only ever removed everything but if you have files in their that enable communication between nodes then keep them. There is likely a less blunt of way handling this but keep in mind you have a secondary to and primary to restore from.

9

Dude, you’re amazing! Thanks for the help, that’s exactly what did it.

For future searchers, the move is to delete eerything in the data directory EXCEPT the keyfile. that needs to stay put.

But once the mongo node showed as Secondary I was able to restart graylog and the log shows port 9000 is listening.

Any idea why this happens? I thought the whole idea behind the cluster for Mogo is that it stops things like this from happening?

1 Like
10

I can’t say I do, interesting that it only becomes an issue at the point of an update to a new major version. I’m assuming this node had been in the recovering state for a while and any service restarts of Graylog and minor bug fix updates didn’t have this same preflight issue.

Probably a good idea to have an NMS monitoring the status of these nodes within the replica set to get a heads up when one isn’t healthy.

11

IMHO for some of the migrations (usually happening during an update/upgrade) we change data in MongoDB and need the replica set in green state. Otherwise the update operation just waits/hangs. Graylog has not started successfully until you see the corresponding “has started” message in the logs.

1 Like