Graylog bootup trouble

Graylog Central (peer support)
1

I upgraded both my graylog nodes from 3.2.4 to 3.2.5 through yum.
Using open jdk 1.8.0_252
I rebooted the servers and while they the startup sequence looks normal, until I get this error:

2020-05-21T01:22:43.676+02:00 INFO  [cluster] Exception in monitor thread while connecting to server rtce-log-mongo2/graylog2:27017
com.mongodb.MongoSocketException: rtce-log-mongo2/graylog2: Name or service not known
        at com.mongodb.ServerAddress.getSocketAddress(ServerAddress.java:188) ~[graylog.jar:?]
        at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:59) ~[graylog.jar:?]
        at com.mongodb.connection.SocketStream.open(SocketStream.java:57) ~[graylog.jar:?]
        at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:126) ~[graylog
.jar:?]
        at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:1
14) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.net.UnknownHostException: rtce-log-mongo2/graylog2: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_252]
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_252]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_252]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_252]
        at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_252]
        at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_252]
        at java.net.InetAddress.getByName(InetAddress.java:1077) ~[?:1.8.0_252]
        at com.mongodb.ServerAddress.getSocketAddress(ServerAddress.java:186) ~[graylog.jar:?]

I then get repeated lines of this:

2020-05-21T01:22:49.052+02:00 INFO [connection] Opened connection [connectionId{localValue:13, serverValue:1488}] to rtce-log-mongo1.unog.un.org:27017
2020-05-21T01:22:49.415+02:00 INFO [connection] Opened connection [connectionId{localValue:14, serverValue:1489}] to rtce-log-mongo1.unog.un.org:27017
2020-05-21T01:22:49.737+02:00 INFO [connection] Opened connection [connectionId{localValue:15, serverValue:1490}] to rtce-log-mongo1.unog.un.org:27017
2020-05-21T01:22:50.059+02:00 INFO [connection] Opened connection [connectionId{localValue:16, serverValue:1491}] to rtce-log-mongo1.unog.un.org:27017
2020-05-21T01:22:50.342+02:00 INFO [connection] Opened connection [connectionId{localValue:17, serverValue:1492}] to rtce-log-mongo1.unog.un.org:27017

The boot sequence does not go any further than this.

Ports 9000, 9200 do not function and the web interface does not come up.

The primary mongodb node logs say that the servers are making connections:

2020-05-21T01:22:49.721+0200 I NETWORK [conn1490] received client metadata from 10.18.221.212:59238 conn1490: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “3.10.0-1127.8.2.el7.x86_64” }, platform: “Java/Oracle Corporation/1.8.0_252-b09” }
2020-05-21T01:22:49.997+0200 I NETWORK [listener] connection accepted from 10.18.221.212:59240 #1491 (24 connections now open)
2020-05-21T01:22:50.026+0200 I NETWORK [conn1491] received client metadata from 10.18.221.212:59240 conn1491: { driver: { name: “mongo-java-driver”, version: “unknown” }, os: { type: “Linux”, name: “Linux”, architecture: “amd64”, version: “3.10.0-1127.8.2.el7.x86_64” }, platform: “Java/Oracle Corporation/1.8.0_252-b09” }

Any help would be appreciated.

2

Can you resolve the hostname?
Mongo started well?
What is your mongo’s cluster state?

3

The hostname is resolvable and the mongo replica set is up and stable.

graylog:PRIMARY> rs.status()
{
“set” : “graylog”,
“date” : ISODate(“2020-05-23T07:28:07.946Z”),
“myState” : 1,
“term” : NumberLong(68),
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“heartbeatIntervalMillis” : NumberLong(2000),
“optimes” : {
“lastCommittedOpTime” : {
“ts” : Timestamp(1590218887, 14),
“t” : NumberLong(68)
},
“readConcernMajorityOpTime” : {
“ts” : Timestamp(1590218887, 14),
“t” : NumberLong(68)
},
“appliedOpTime” : {
“ts” : Timestamp(1590218887, 14),
“t” : NumberLong(68)
},
“durableOpTime” : {
“ts” : Timestamp(1590218887, 14),
“t” : NumberLong(68)
}
},
“lastStableCheckpointTimestamp” : Timestamp(1590218876, 14),
“electionCandidateMetrics” : {
“lastElectionReason” : “stepUpRequestSkipDryRun”,
“lastElectionDate” : ISODate(“2020-05-20T21:43:23.884Z”),
“electionTerm” : NumberLong(68),
“lastCommittedOpTimeAtElection” : {
“ts” : Timestamp(1590011003, 1),
“t” : NumberLong(67)
},
“lastSeenOpTimeAtElection” : {
“ts” : Timestamp(1590011003, 1),
“t” : NumberLong(67)
},
“numVotesNeeded” : 2,
“priorityAtElection” : 1,
“electionTimeoutMillis” : NumberLong(10000),
“priorPrimaryMemberId” : 2,
“numCatchUpOps” : NumberLong(0),
“newTermStartDate” : ISODate(“2020-05-20T21:43:23.896Z”),
“wMajorityWriteAvailabilityDate” : ISODate(“2020-05-20T21:43:25. 016Z”)
},
“members” : [
{
“_id” : 1,
“name” : “rtce-log-mongo1.unog.un.org:27017”,
“health” : 1,
“state” : 1,
“stateStr” : “PRIMARY”,
“uptime” : 208951,
“optime” : {
“ts” : Timestamp(1590218887, 14),
“t” : NumberLong(68)
},
“optimeDate” : ISODate(“2020-05-23T07:28:07Z”),
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“infoMessage” : “”,
“electionTime” : Timestamp(1590011003, 2),
“electionDate” : ISODate(“2020-05-20T21:43:23Z”),
“configVersion” : 10,
“self” : true,
“lastHeartbeatMessage” : “”
},
{
“_id” : 2,
“name” : “rtce-log-mongo2.unog.un.org:27017”,
“health” : 1,
“state” : 2,
“stateStr” : “SECONDARY”,
“uptime” : 206227,
“optime” : {
“ts” : Timestamp(1590218886, 6),
“t” : NumberLong(68)
},
“optimeDurable” : {
“ts” : Timestamp(1590218886, 6),
“t” : NumberLong(68)
},
“optimeDate” : ISODate(“2020-05-23T07:28:06Z”),
“optimeDurableDate” : ISODate(“2020-05-23T07:28:06Z”),
“lastHeartbeat” : ISODate(“2020-05-23T07:28:06.882Z”),
“lastHeartbeatRecv” : ISODate(“2020-05-23T07:28:06.220Z” ),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “”,
“syncingTo” : “rtce-log-mongo1.unog.un.org:27017”,
“syncSourceHost” : “rtce-log-mongo1.unog.un.org:27017”,
“syncSourceId” : 1,
“infoMessage” : “”,
“configVersion” : 10
},
{
“_id” : 3,
“name” : “rtce-log-mongo3.unog.un.org:27017”,
“health” : 1,
“state” : 2,
“stateStr” : “SECONDARY”,
“uptime” : 206137,
“optime” : {
“ts” : Timestamp(1590218886, 7),
“t” : NumberLong(68)
},
“optimeDurable” : {
“ts” : Timestamp(1590218886, 7),
“t” : NumberLong(68)
},
“optimeDate” : ISODate(“2020-05-23T07:28:06Z”),
“optimeDurableDate” : ISODate(“2020-05-23T07:28:06Z”),
“lastHeartbeat” : ISODate(“2020-05-23T07:28:06.910Z”),
“lastHeartbeatRecv” : ISODate(“2020-05-23T07:28:06.063Z” ),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “”,
“syncingTo” : “rtce-log-mongo1.unog.un.org:27017”,
“syncSourceHost” : “rtce-log-mongo1.unog.un.org:27017”,
“syncSourceId” : 1,
“infoMessage” : “”,
“configVersion” : 10
}
],
“ok” : 1,
“operationTime” : Timestamp(1590218887, 14),
“$clusterTime” : {
“clusterTime” : Timestamp(1590218887, 14),
“signature” : {
“hash” : BinData(0,“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”),
“keyId” : NumberLong(0)
}
}
}

4

in this case continue the throubleshooting,
database name, username, password; firewall, etc

5

The database, credentials are the same ones from 3.2.4 and it functions correctly.

The firewall rules allow for connection.
The logs on mongodb indicate that there is connection.
The only thing that has changed is the upgrade to 3.2.5.

There are release notes stating that the mongodb driver has changed for 3.3.0
Has there been a similar change in 3.2.5?

In the meantime, I’ve reverted back to 3.2.4 and it functions correctly.

Please advise.

6

I’m just an user, so I have no more idea.
Maybe some developers can help in the mongo driver changes. (But I think it should in the release notes.)

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.