Multi Node Graylog Setup problem

Mbuyukkarakas · March 6, 2019, 10:34am

Dear all, I’m trying to build a 3 node GL cluster but have some strange logs and cant bring it up.

My cluster is using GL 2.5 / EL 5.5 / Mongo 4.0.6. No firewall, no SELinux

IPs :
node01.local 172.23.88.9
node02.local 172.23.88.10
node03.local 172.23.88.11

Mongo DB / Primary Config

rs0:PRIMARY> rs.config()
{
“_id” : “rs0”,
“version” : 3,
“protocolVersion” : NumberLong(1),
“writeConcernMajorityJournalDefault” : true,
“members” : [
{
“_id” : 0,
“host” : “node01.local:27017”,
“arbiterOnly” : false,
“buildIndexes” : true,
“hidden” : false,
“priority” : 1,
“tags” : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 1,
		"host" : "node02.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : false,
		"priority" : 1,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	},
	{
		"_id" : 2,
		"host" : "node03.local:27017",
		"arbiterOnly" : false,
		"buildIndexes" : true,
		"hidden" : false,
		"priority" : 1,
		"tags" : {

		},
		"slaveDelay" : NumberLong(0),
		"votes" : 1
	}
],
"settings" : {
	"chainingAllowed" : true,
	"heartbeatIntervalMillis" : 2000,
	"heartbeatTimeoutSecs" : 10,
	"electionTimeoutMillis" : 10000,
	"catchUpTimeoutMillis" : -1,
	"catchUpTakeoverDelayMillis" : 30000,
	"getLastErrorModes" : {

	},
	"getLastErrorDefaults" : {
		"w" : 1,
		"wtimeout" : 0
	},
	"replicaSetId" : ObjectId("5c7f9b9d2cd4ee245aee4df5")
}

}

MongoDB Secondary server view :

rs0:SECONDARY> rs.slaveOk()
rs0:SECONDARY> db.printSlaveReplicationInfo()
source: node02.local:27017
syncedTo: Wed Mar 06 2019 13:28:31 GMT+0300
-1 secs (0 hrs) behind the primary
source: node03.local:27017
syncedTo: Wed Mar 06 2019 13:28:30 GMT+0300
0 secs (0 hrs) behind the primary
rs0:SECONDARY> show dbs
admin 0.000GB
config 0.000GB
graylog 0.001GB
local 0.000GB

graylog.conf file

…
rest_listen_uri = http://node01.local:9000/api/
rest_transport_uri = http://node01.local:9000/api/
web_listen_uri = http://node01.local:9000/
elasticsearch_hosts = http://node01.local:9200,http://node02.local:9200,http://node03.local:9200
#mongodb://graylog:123456@node01.local:27017,node02.local:27017,node03.local:27017/graylog?replicaSet=rs0
mongodb://node01.local:27017,node02.local:27017,node03.local:27017/graylog?replicaSet=rs0
…

graylog node 01 graylog.log file

019-03-06T13:11:19.455+03:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2019-03-06T13:11:23.842+03:00 INFO [InputBufferImpl] Message journal is enabled.
2019-03-06T13:11:23.879+03:00 INFO [NodeId] Node ID: 8f49d4e6-5d90-4c10-906f-7324ad14c678
2019-03-06T13:11:24.245+03:00 INFO [LogManager] Loading logs.
2019-03-06T13:11:24.375+03:00 INFO [LogManager] Logs loading complete.
2019-03-06T13:11:24.376+03:00 INFO [KafkaJournal] Initialized Kafka based journal at /data/graylog/journal
2019-03-06T13:11:24.408+03:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2019-03-06T13:11:24.449+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2019-03-06T13:11:24.541+03:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2019-03-06T13:11:24.838+03:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:20}] to localhost:27017
2019-03-06T13:11:24.854+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 6]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=3674710, setName=‘rs0’, canonicalAddress=node01.local:27017, hosts=[node02.local:27017, node03.local:27017, node01.local:27017], passives=, arbiters=, primary=‘node01.local:27017’, tagSet=TagSet{}, electionId=7fffffff0000000000000001, setVersion=3, lastWriteDate=Wed Mar 06 13:11:23 EET 2019, lastUpdateTimeNanos=11093256202569}
2019-03-06T13:11:24.870+03:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:21}] to localhost:27017
…
…

node02 graylog.log file

2019-03-06T13:13:59.534+03:00 INFO [LogManager] Logs loading complete.
2019-03-06T13:13:59.534+03:00 INFO [KafkaJournal] Initialized Kafka based journal at /data/graylog/journal
2019-03-06T13:13:59.573+03:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2019-03-06T13:13:59.608+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2019-03-06T13:13:59.701+03:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2019-03-06T13:13:59.743+03:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:25}] to localhost:27017
2019-03-06T13:13:59.758+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 6]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=3617725, setName=‘rs0’, canonicalAddress=node02.local:27017, hosts=[node02.local:27017, node03.local:27017, node01.local:27017], passives=, arbiters=, primary=‘node01.local:27017’, tagSet=TagSet{}, electionId=null, setVersion=3, lastWriteDate=Wed Mar 06 13:13:58 EET 2019, lastUpdateTimeNanos=6793303396622}
2019-03-06T13:13:59.797+03:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:26}] to localhost:27017
2019-03-06T13:14:00.178+03:00 INFO [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 6]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=3617725, setName=‘rs0’, canonicalAddress=node02.local:27017, hosts=[node02.local:27017, node03.local:27017, node01.local:27017], passives=, arbiters=, primary=‘node01.local:27017’, tagSet=TagSet{}, electionId=null, setVersion=3, lastWriteDate=Wed Mar 06 13:13:58 EET 2019, lastUpdateTimeNanos=6793303396622}]}. Waiting for 30000 ms before timing out
2019-03-06T13:14:30.192+03:00 INFO [cluster] No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 6]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=1199113, setName=‘rs0’, canonicalAddress=node02.local:27017, hosts=[node02.local:27017, node03.local:27017, node01.local:27017], passives=, arbiters=, primary=‘node01.local:27017’, tagSet=TagSet{}, electionId=null, setVersion=3, lastWriteDate=Wed Mar 06 13:14:28 EET 2019, lastUpdateTimeNanos=6823420143388}]}. Waiting for 30000 ms before timing out

My questions are ;

Why cant make it up ?
Why I’m always seeing in the log files that the mongo is connecting locally instead of cluster on all of my servers ?

Any help ?
Thank you.

Mehmet

Mbuyukkarakas · March 6, 2019, 12:01pm

Dear all, I’m trully sorry to take your time. I have found the problem.

My graylog.conf file has this line ;

mongodb://node01.local:27017,node02.local:27017,node03.local:27017/graylog?replicaSet=rs0

Actually it should be like this ;

mongodb_uri = mongodb://node01.local:27017,node02.local:27017,node03.local:27017/graylog?replicaSet=rs0

I forgot to add “mongodb_uri =” part. Copy / Paste issues from text editor

Best regards.
Mehmet

system · March 20, 2019, 12:05pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Multi node Graylog cluster not connecting to mongodb Graylog Central (peer support)	2	525	September 15, 2019
Only 1 node showing in Graylog Web Interface Graylog Central (peer support)	5	557	May 26, 2023
Only 1 of 3 nodes shows gl configuration after upgrade Graylog Central (peer support)	3	638	October 6, 2020
How to setup multi node in graylog manually Graylog Central (peer support)	2	5635	February 15, 2018
Multi nodes question Graylog Central (peer support)	4	1477	April 16, 2019