Mongo_uri set correctly I think but graylog says ERROR [cluster] Expecting replica set member from set

[log]

1. Describe your incident:
Everything looks nice:
2023-03-08T13:41:01.063-05:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=FIRST:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=20604162, setName='rs0', canonicalAddress=PRIMARYGUY:27017, hosts=[SECONDARY1:27021, PRIMARYGUY:27017, SECONDARY2:27022], passives=[], arbiters=[], primary='SECONDARY1:27021', tagSet=TagSet{[]}, electionId=null, setVersion=5, topologyVersion=TopologyVersion{processId=6408d689b8c28e96ad630057, counter=4}, lastWriteDate=Wed Mar 08 13:40:52 EST 2023, lastUpdateTimeNanos=11421634809451}

And then it doesnt:
2023-03-08T13:41:01.065-05:00 ERROR [cluster] Expecting replica set member from set 'rs0,mongodb://SECONDARY1:27021/graylog?replicaSet=rs0,mongodb://SECONDARY2:27022/graylog?replicaSet=rs0', but found one from set 'rs0'. Removing PRIMARYGUY:27017 from client view of cluster.

I need help interpreting whats going on here.

2. Describe your environment:

• OS Information:
  Rocky 8

• Package Version:
  graylog-server-5.0.2-1.x86_64
  mongodb-org-server-6.0.4-1.el8.x86_64

• Service logs, configurations, and environment variables:

Here’s how I started all my mongos:

mongod --port 27017 --dbpath /var/lib/mongo --replSet rs0 --bind_ip PRIMARYGUY
mongod --port 27021 --dbpath /var/lib/mongo --replSet rs0 --bind_ip SECONDARY1
mongod --port 27022 --dbpath /var/lib/mongo --replSet rs0 --bind_ip SECONDARY2

Heres my graylog server.conf mongo uri line for replicas:

# Use a replica set instead of a single host
mongodb_uri = mongodb://PRIMARYGUY:27017/graylog?replicaSet=rs0,mongodb://SECONDARY1:27021/graylog?replicaSet=rs0,mongodb://SECONDARY2:27022/graylog?replicaSet=rs0

and tried:

# Use a replica set instead of a single host
mongodb_uri = mongodb://PRIMARYGUY:27017/graylog?replicaSet=rs0,SECONDARY1:27021/graylog?replicaSet=rs0,SECONDARY2:27022/graylog?replicaSet=rs0

And mongosh’d into PRIMARYGUY shows the set is named rs0… and that all 3 buddies are members!

3. What steps have you already taken to try and solve the problem?
I tried all varieties of mongo uri lines (removed mongodb:// and remoed the replicaSet param…), but graylog sometimes starts up and sometimes doesnt, and at any rate, delivers a new error around the mongos.

4. How can the community help?

Mainly I need to know why “Removing PRIMARYGUY:27017 from client view of cluster.” this is happening after it appears to start up successfully knowing the replica set name.

Am I not supposed to specify “/graylog?replicaSet=rs0” for every server or something?

[gsmith]

Hey @log

First, thanks for making me smile today, PRIMARYGUY

Your settings look correct, I see you using Differen port for each mondoDb, but did you configure each Mongo node with those ports?

vi /etc/mongod.conf

net:
  port: 27017
  bindIp: PRIMARYGUY
replication:
  replicaSetName: mongoReplicaSet

net:
  port: 27021
  bindIp: SECONDARY1
replication:
  replicaSetName: mongoReplicaSet

net:
  port: 27022
  bindIp: SECONDARY2
replication:
  replicaSetName: mongoReplicaSet

Second, when you execute this mongo shell command , what do you see?

mongoReplicaSet:PRIMARYGUY> rs.config()
mongoReplicaSet:PRIMARYGUY> rs.status()

Here is my config output

{
        "_id" : "mongoReplicaSet",
        "version" : 3,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongo-node-1.domain.com:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "mongo-node-2.domain.com:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "mongo-node-3.domain.com:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5d1e36618481a2906eb97171")
        }
}

Here is my Status output

# Example output
mongoReplicaSet:PRIMARY> rs.status()
{
        "set" : "mongoReplicaSet",
        "date" : ISODate("2019-07-10T21:58:51.943Z"),
        "myState" : 1,
        "term" : NumberLong(11),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1562795931, 2),
                        "t" : NumberLong(11)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1562795931, 2),
                        "t" : NumberLong(11)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1562795931, 2),
                        "t" : NumberLong(11)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1562795931, 2),
                        "t" : NumberLong(11)
                }
        },
        "lastStableCheckpointTimestamp" : Timestamp(1562795908, 3),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongo1.domain.net:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 5542,
                        "optime" : {
                                "ts" : Timestamp(1562795931, 1),
                                "t" : NumberLong(11)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1562795931, 1),
                                "t" : NumberLong(11)
                        },
                        "optimeDate" : ISODate("2019-07-10T21:58:51Z"),
                        "optimeDurableDate" : ISODate("2019-07-10T21:58:51Z"),
                        "lastHeartbeat" : ISODate("2019-07-10T21:58:51.622Z"),
                        "lastHeartbeatRecv" : ISODate("2019-07-10T21:58:50.303Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "gl-mongo2.domain.com:27017",
                        "syncSourceHost" : "gl-mongo2.domain.com:27017",
                        "syncSourceId" : 1,
                        "infoMessage" : "",
                        "configVersion" : 3
                },
                {
                        "_id" : 1,
                        "name" : "mongo2.domain.net:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 5669,
                        "optime" : {
                                "ts" : Timestamp(1562795931, 2),
                                "t" : NumberLong(11)
                        },
                        "optimeDate" : ISODate("2019-07-10T21:58:51Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1562790394, 1),
                        "electionDate" : ISODate("2019-07-10T20:26:34Z"),
                        "configVersion" : 3,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 2,
                        "name" : "mongo3.domain.net:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 5511,
                        "optime" : {
                                "ts" : Timestamp(1562795931, 1),
                                "t" : NumberLong(11)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1562795931, 1),
                                "t" : NumberLong(11)
                        },
                        "optimeDate" : ISODate("2019-07-10T21:58:51Z"),
                        "optimeDurableDate" : ISODate("2019-07-10T21:58:51Z"),
                        "lastHeartbeat" : ISODate("2019-07-10T21:58:51.686Z"),
                        "lastHeartbeatRecv" : ISODate("2019-07-10T21:58:50.938Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "mongo2.enseva-labs.net:27017",
                        "syncSourceHost" : "mongo2.domain.com:27017",
                        "syncSourceId" : 1,
                        "infoMessage" : "",
                        "configVersion" : 3
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1562795931, 2),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1562795931, 2),
                "signature" : {
                        "hash" : BinData(0,"K2p6/W2VrP3eycBcW9xzneDo9Gc="),
                        "keyId" : NumberLong("6709860293658279938")
                }
        }
}

EDIT:
I forgot to ask , when you created Mongo cluster did you go through Mongo shell
Example, this is from my PRIMARYGUY.

# Enter the following command to create the replica set.
> rs.initiate()

# Enter the following command to add other nodes, where the other node is named, 
# for example, "mongo-node-2" and the domain is domain.com.
> rs.add("mongo-node-2.domain.com")
> rs.initiate()
# Enter the following command to add other nodes, where the other node is named, 
# for example, "mongo-node-3" and the domain is domain.com.
> rs.add("mongo-node-3.domain.com")

Connecting to MongoDB replica set results in "Expecting replica set member, but found a STANDALONE"