After upgrading from 4.2.x to 4.3.3, on Debian 10.1, my leader node will no longer join the cluster. Graylog server.log shows an exception trying to connect to MongoDB, but everything looks OK with Mongo. Time is synced. Mongo URI is consistent between all nodes.
I did resolve a mongoDB error where it couldn’t access/write to the /tmp/mongodb-27017.sock, but deleting /tmp/mongodb-27017.sock and restarting mongoDB seemed to clear that up.
This is a 3 node cluster (plus load balancer and frontend).
Server.log shows:
2022-08-02T13:51:02.746-08:00 INFO [cluster] Adding discovered server GrayLogServerName2:27017 to client view of cluster
2022-08-02T13:51:02.788-08:00 INFO [cluster] Adding discovered server GrayLogServerName3:27017 to client view of cluster
2022-08-02T13:51:02.789-08:00 INFO [cluster] Adding discovered server GrayLogServerName4:27017 to client view of cluster
2022-08-02T13:51:02.844-08:00 INFO [cluster] No server chosen by com.mongodb.client.internal.MongoClientDelegate$1@1db7157f from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=GrayLogServerName2:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=GrayLogServerName3:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=GrayLogServerName4:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2022-08-02T13:51:02.823-08:00 INFO [cluster] Exception in monitor thread while connecting to server GrayLogServerName4:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[graylog.jar:?]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[graylog.jar:?]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[graylog.jar:?]
… 3 more
2022-08-02T13:51:02.866-08:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:289}] to GrayLogServerName3:27017
2022-08-02T13:51:02.868-08:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:1687}] to GrayLogServerName2:27017
Can anyone advise on troubleshooting Graylog not connecting to MongoDB when ‘service mongod status’ shows mongod as running?