Unable to connect graylog cluster with mongodb replicaset

I am trying to run graylog production setup (3 mongo nodes replica set and elastic cluster having 3 nodes).

  1. Graylog version - graylog/graylog:2.3.0-1 (docker)
  2. mongodb version - mongo:3.6 (docker)

I have used following mongodb configurations for graylog config:

mongodb_uri = mongodb://username:password@mongo2.xxxx.com:27017,mongo3.xxxx.com:27017,mongo1.xxxx.com:27017/graylog?replicaSet=rset
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5

Mongodb cluster is running without any issue. But graylog containers are terminating by throwing following error:

2018-02-22 14:23:24,843 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.3.0 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-02-22 14:23:24,847 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.3.0 [org.graylog.plugins.collector.CollectorPlugin]
2018-02-22 14:23:24,848 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.3.0 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-02-22 14:23:24,849 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.3.0 [org.graylog.plugins.map.MapWidgetPlugin]
2018-02-22 14:23:24,860 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.3.0 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-02-22 14:23:24,861 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.3.0 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2018-02-22 14:23:25,186 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2018-02-22 14:23:25,558 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2018-02-22 14:23:28,170 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2018-02-22 14:23:28,195 INFO : org.graylog2.plugin.system.NodeId - Node ID: 2c7af0bc-95f0-4df7-b1a3-cb614e23a96a
2018-02-22 14:23:28,377 INFO : kafka.log.LogManager - Loading logs.
2018-02-22 14:23:28,397 WARN : kafka.log.Log - Found a corrupted index file, /usr/share/graylog/data/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2018-02-22 14:23:28,418 INFO : kafka.log.LogManager - Logs loading complete.
2018-02-22 14:23:28,418 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2018-02-22 14:23:28,430 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-02-22 14:23:28,446 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo1.xxxx.com:27017, mongo2.xxxx.com:27017, mongo3.xxxx.com:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500, requiredReplicaSetName='rset'}
2018-02-22 14:23:28,446 INFO : org.mongodb.driver.cluster - Adding discovered server mongo1.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,470 INFO : org.mongodb.driver.cluster - Adding discovered server mongo2.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,476 INFO : org.mongodb.driver.cluster - Adding discovered server mongo3.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,489 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-02-22 14:23:28,529 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo1.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:28,556 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo3.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:28,556 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo2.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:58,491 ERROR: org.graylog2.bindings.providers.MongoConnectionProvider - Error connecting to MongoDB: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:23:58,775 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:24:28,799 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:24:58,826 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:25:28,839 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:25:30,023 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:694, serverValue:402}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,481 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 3 servers: [http://elastic:devops@192.168.35.139:9200,http://elastic:devops@192.168.35.140:9200,http://elastic:devops@192.168.35.141:9200]
2018-02-22 14:25:30,482 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Authentication cache set for preemptive authentication
2018-02-22 14:25:30,963 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:32,855 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2018-02-22 14:25:33,034 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,044 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:33,088 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,132 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,172 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,212 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,647 ERROR: org.graylog2.commands.Server - 

################################################################################

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

Did the user you use has the permissions to access like described in the docs?

http://docs.graylog.org/en/2.4/pages/configuration/multinode_setup.html#mongodb-replica-set

Are you sure that the MongoDB nodes listed in mongodb_uri are accessible for Graylog on port 27017/tcp?

The error message indicates that either some firewall is blocking access or that the MongoDB daemon isn’t listening on the public network interface of these machines.

2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}

2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,481 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 3 servers: [http://elastic:devops@192.168.35.139:9200,http://elastic:devops@192.168.35.140:9200,http://elastic:devops@192.168.35.141:9200]
2018-02-22 14:25:30,482 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Authentication cache set for preemptive authentication
2018-02-22 14:25:30,963 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:32,855 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2018-02-22 14:25:33,034 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,044 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:33,088 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,132 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,172 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,212 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Doesn’t it say that that graylog was connected to the primary replica. But afterwards it fails to connect.

yes, this user has the roles readWrite and dbAdmin.

2018-02-22 14:25:30,023 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:694, serverValue:402}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017

it seems it was connected to mongoDB instance. Afterwards it throws the error

What’s the output of the following commands?

# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo1.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo2.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo3.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo2.xxxx.com:27017,mongo3.xxxx.com:27017,mongo1.xxxx.com:27017/graylog?replicaSet=rset'

First i was trying this by adding mongo hosts in /etc/hosts lists of graylog container, but the graylog container was exiting after a moment. Then i tried to ran the containers by using ‘–add host’ variable while starting the graylog container. Then graylog container was not terminating. Now it works fine.

Thank you each and everyone who put such an effort to help me…
Thanks again…

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.