Unable to connect graylog cluster with mongodb replicaset


(Vaseemt) #1

I am trying to run graylog production setup (3 mongo nodes replica set and elastic cluster having 3 nodes).

  1. Graylog version - graylog/graylog:2.3.0-1 (docker)
  2. mongodb version - mongo:3.6 (docker)

I have used following mongodb configurations for graylog config:

mongodb_uri = mongodb://username:password@mongo2.xxxx.com:27017,mongo3.xxxx.com:27017,mongo1.xxxx.com:27017/graylog?replicaSet=rset
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5

Mongodb cluster is running without any issue. But graylog containers are terminating by throwing following error:

2018-02-22 14:23:24,843 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.3.0 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-02-22 14:23:24,847 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.3.0 [org.graylog.plugins.collector.CollectorPlugin]
2018-02-22 14:23:24,848 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.3.0 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-02-22 14:23:24,849 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.3.0 [org.graylog.plugins.map.MapWidgetPlugin]
2018-02-22 14:23:24,860 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.3.0 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-02-22 14:23:24,861 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.3.0 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2018-02-22 14:23:25,186 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2018-02-22 14:23:25,558 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2018-02-22 14:23:28,170 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2018-02-22 14:23:28,195 INFO : org.graylog2.plugin.system.NodeId - Node ID: 2c7af0bc-95f0-4df7-b1a3-cb614e23a96a
2018-02-22 14:23:28,377 INFO : kafka.log.LogManager - Loading logs.
2018-02-22 14:23:28,397 WARN : kafka.log.Log - Found a corrupted index file, /usr/share/graylog/data/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index...
2018-02-22 14:23:28,418 INFO : kafka.log.LogManager - Logs loading complete.
2018-02-22 14:23:28,418 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2018-02-22 14:23:28,430 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-02-22 14:23:28,446 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo1.xxxx.com:27017, mongo2.xxxx.com:27017, mongo3.xxxx.com:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500, requiredReplicaSetName='rset'}
2018-02-22 14:23:28,446 INFO : org.mongodb.driver.cluster - Adding discovered server mongo1.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,470 INFO : org.mongodb.driver.cluster - Adding discovered server mongo2.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,476 INFO : org.mongodb.driver.cluster - Adding discovered server mongo3.xxxx.com:27017 to client view of cluster
2018-02-22 14:23:28,489 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-02-22 14:23:28,529 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo1.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:28,556 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo3.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:28,556 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo2.xxxx.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[graylog.jar:?]
	at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131]
	at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[graylog.jar:?]
	at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[graylog.jar:?]
	... 3 more
2018-02-22 14:23:58,491 ERROR: org.graylog2.bindings.providers.MongoConnectionProvider - Error connecting to MongoDB: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:23:58,775 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:24:28,799 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:24:58,826 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:25:28,839 INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out
2018-02-22 14:25:30,023 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:694, serverValue:402}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,481 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 3 servers: [http://elastic:devops@192.168.35.139:9200,http://elastic:devops@192.168.35.140:9200,http://elastic:devops@192.168.35.141:9200]
2018-02-22 14:25:30,482 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Authentication cache set for preemptive authentication
2018-02-22 14:25:30,963 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:32,855 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2018-02-22 14:25:33,034 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,044 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:33,088 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,132 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,172 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,212 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,647 ERROR: org.graylog2.commands.Server - 

################################################################################

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(


(Jan Doberstein) #2

Did the user you use has the permissions to access like described in the docs?

http://docs.graylog.org/en/2.4/pages/configuration/multinode_setup.html#mongodb-replica-set


(Jochen) #3

Are you sure that the MongoDB nodes listed in mongodb_uri are accessible for Graylog on port 27017/tcp?

The error message indicates that either some firewall is blocking access or that the MongoDB daemon isn’t listening on the public network interface of these machines.


(Mushycode) #4

2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}

2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,481 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 3 servers: [http://elastic:devops@192.168.35.139:9200,http://elastic:devops@192.168.35.140:9200,http://elastic:devops@192.168.35.141:9200]
2018-02-22 14:25:30,482 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2018-02-22 14:25:30,556 INFO : io.searchbox.client.JestClientFactory - Authentication cache set for preemptive authentication
2018-02-22 14:25:30,963 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:32,855 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2018-02-22 14:25:33,034 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,044 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-02-22 14:25:33,088 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,132 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,172 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,212 WARN : org.graylog.plugins.map.geoip.GeoIpResolverEngine - GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,645 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
2018-02-22 14:25:33,646 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo3.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo1.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=mongo2.xxxx.com:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

Doesn’t it say that that graylog was connected to the primary replica. But afterwards it fails to connect.


(Vaseemt) #5

yes, this user has the roles readWrite and dbAdmin.


(Vaseemt) #6
2018-02-22 14:25:30,023 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:694, serverValue:402}] to mongo1.xxxx.com:27017
2018-02-22 14:25:30,035 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo1.xxxx.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 2]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=3337194, setName='rset', canonicalAddress=mongo1.xxxx.com:27017, hosts=[mongo2.xxxx.com:27017, mongo1.xxxx.com:27017, mongo3.xxxx.com:27017], passives=[], arbiters=[], primary='mongo1.xxxx.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=4, lastWriteDate=Thu Feb 22 14:25:21 UTC 2018, lastUpdateTimeNanos=617671104955137}
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max election id to 7fffffff0000000000000001 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Setting max set version to 4 from replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,036 INFO : org.mongodb.driver.cluster - Discovered replica set primary mongo1.xxxx.com:27017
2018-02-22 14:25:30,118 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:696, serverValue:403}] to mongo1.xxxx.com:27017

it seems it was connected to mongoDB instance. Afterwards it throws the error


(Jochen) #7

What’s the output of the following commands?

# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo1.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo2.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo3.xxxx.com:27017/graylog'
# echo '{ ping: 1 }' | mongo 'mongodb://username:password@mongo2.xxxx.com:27017,mongo3.xxxx.com:27017,mongo1.xxxx.com:27017/graylog?replicaSet=rset'

(Vaseemt) #8

First i was trying this by adding mongo hosts in /etc/hosts lists of graylog container, but the graylog container was exiting after a moment. Then i tried to ran the containers by using ‘–add host’ variable while starting the graylog container. Then graylog container was not terminating. Now it works fine.


(Vaseemt) #9

Thank you each and everyone who put such an effort to help me…
Thanks again…


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.