Graylog docker did not connect to MongoDB docker

Graylog Central (peer support)
1

I’m trying to connect a 2-server Graylog docker in a MongoDB replica set, also in docker. MongoDB replica set is working OK.

IMAGE               PORTS                                  NAMES
mongo               0.0.0.0:27017-27019->27017-27019/tcp   mongo

[ReplicaSetMonitor-TaskExecutor] Successfully connected to server-1:27017 (1 connections now open to server-1:27017 with a 5 second timeout)  
[LogicalSessionCacheRefresh] Successfully connected to server-2:27017 (1 connections now open to server-2:27017 with a 5 second timeout)  
[ReplicaSetMonitor-TaskExecutor] Successfully connected to server-3:27017 (1 connections now open to server-3:27017 with a 5 second timeout)  
Starting new replica set monitor for graylog_prd_1/server-2:27017,server-3:27017,server-1:27017

But, when I execute a docker run command, it didn’t find the mongo.

sudo docker run -d -e TZ=America/Sao_Paulo --restart always --name graylog -p 80:9000 -p 514:514 -p 514:514/udp -p 12201:12201 -p 12201:12201/udp -v /opt/open/graylog:/usr/share/graylog graylog/graylog:2.5 && sudo docker logs -f graylog &

Cluster created with settings {hosts=[mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=mongo:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
Exception in monitor thread while connecting to server mongo:27017 com.mongodb.MongoSocketException: mongo: Name or service not known

I’ve tried to use all 3 config modes in the graylog.conf, but none of them seens to work.

# MongoDB connection string
mongodb_uri = mongodb://server-1:27017/graylog_prd_1

# Authenticate against the MongoDB server
mongodb_uri = mongodb://admin:graylog@server-1:27017/graylog_prd_1

# Use a replica set instead of a single host
mongodb_uri = mongodb://admin:graylog@server-2,server-3,server-1:27017/?replicaSet=graylog_prd_1

Seens that Graylog is trying to connect to a local MongoDB. Have I missed anything in docker run command that allows the connection?
I’ve followed the instructions from MongoDB site and Graylog site, and have searched hundreds of sites, but nothing solve my problem.
I can connect to MongoDB via telnet,

[user@server-2 ~]$ telnet localhost 27017
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

[user@server-2 ~]$ telnet 10.216.76.183 27017
Trying 10.216.76.183...
Connected to 10.216.76.183.
Escape character is '^]'.

Thanks in advance.

Configuration:
Server-1: MongoDB Service Master
Server-2: MongoDB replica / Graylog Server 1
Server-3: MongoDB replica / Graylog Server 2

2

when you use DNS in your configuration you need to make sure all servers can resolve that in the same way. You are using Names in the configuration and for example IP in your telnet session.

In addition you need to have ONE monodb_uri string and not 3 - in addition I do mot see how you provide the configuration to Graylog. Please read the docker instalaltion manual again

1 Like
3

I have DNS configured in my environment. All servers can access. Those telnet tests were to check if these ports were “listening”. About the mongodb strings, I’ve tried one at a time, not all together.
Graylog ins’t finishing installation because it doesn’t find mongo to connect. So, the configuration ins’t a problem right now.
I have a graylog/mongodb/elasticsearch infrastructure built, but without docker.

4

taking your first mentioned docker command - you did not link the container.

please re-read our documentation: http://docs.graylog.org/en/2.5/pages/installation/docker.html

1 Like
5

Thanks. The command “–link mongo” solved that problem, but I can’t still finish the installation of Graylog.
I’m getting this message:

INFO : org.mongodb.driver.cluster - No server chosen by WritableServerSelector from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=mongo:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 5]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=1091552, setName=‘graylog_prd_1’, canonicalAddress=server-2:27017, hosts=[server-3:27017, server-2:27017, server-1:27017], passives=, arbiters=, primary=‘server-1:27017’, tagSet=TagSet{}, electionId=null, setVersion=3, lastWriteDate=Mon Jan 28 16:48:34 BRST 2019, lastUpdateTimeNanos=1557724478842851}]}. Waiting for 30000 ms before timing out

Seens like I’m connected to replica set, but it stops there. I’ve found this line on log that I could’t solve.

Query failed with error code 13 and error message ‘command find requires authentication’ on server mongo:27017/

My connection string is:

mongodb_uri = mongodb://user:passwd@server-1:27017,server-2:27017,server-3:27017/admin?replicaSet=graylog_prd_1

I create the user with this:

root@node1:/# mongo
use admin
db.createUser( {
user: “admin”,
pwd: “graylog”,
roles: [ { role: “userAdminAnyDatabase”, db: “admin” } ]
});
db.createUser( {
user: “root”,
pwd: “admin20”,
roles: [ { role: “root”, db: “admin” } ]
});

I’ve tried to configure both users, but none allows me to finish the installation.

1 Like
6

Did you build a replicaset in MongoDB?

Without knowing your configuration we can’t help you. In addition please read the Graylog documentation about Docker setups - they contain working configuration.

2 Likes
7

As far as I remember it means no primary node chosen in your mongodb cluster.

2 Likes
8

Yup that sounds very likely.

So either the cluster is unhealthy, or as Jan pointed out the cluster is actually still non-existent :smiley:

1 Like
9

The replica set is built.

Starting new replica set monitor for graylog_sistemas/server-2:27017,server-3:27017,server-1:27017

But, when I execute the command rs.conf() on the primary node, these are the status:

“name” : “server-1:27017”, / “health” : 1, / “state” : 1 / “stateStr” : “PRIMARY”,
“name” : “server-2:27017”, / “health” : 1, / “state” : 2, / “stateStr” : “SECONDARY”,
“name” : “server-3:27017”, / “health” : 1, / “state” : 0, / “stateStr” : “STARTUP”,

The primary and secondary nodes are showing these messages on log all the time:

NETWORK [listener] connection accepted from 10.216.76.138:33460 #36419 (8 connections now open)
ACCESS [conn36419] Successfully authenticated as principal __system on local
NETWORK [conn36419] end connection 10.216.76.138:33460 (7 connections now open)

I’ve tried to reconfig the members, remove e readd the member, clean db directory and restart mongo, but none of then solved. I’ve searched about this, but nothing has worked.

10

I’ve read the documentation many times, but didn’t find the error in my configuration.

I’ve built a replica set: 3 servers - 1 primary and 2 secondarys.
Graylog will be installed in 2 servers, with docker swarm.
I wiil use HAProxy to balance the connections.
Made a Elasticsearch cluster with 3 servers.

11

All three look healthy (“health”:1) and as you said: they’re receiving and accepting connections. So Mongo looks okay in that regard.

The question now is why your system appears to be unable to find a writable node, the primary.

For verification purposes, can you confirm that you can connect to the MongoDB primary node, from your Graylog box? First using Netcat to connect to the port in question, then trying the mongo shell. You should be able to login to MongoDB using the mongo shell, from the Graylog host using the Graylog credentials.

If that also works, time to manually test the db with things like show dbs and possibly an InsertOne()into a dummy table of the Graylog collection.

1 Like
12

I can connect to all 3 servers from each other.

root@server-1:/# mongo --host server-2 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://drjscgerlx042:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“148d025f-8012-4c60-bcfd-3b28c1231ed2”) }
MongoDB server version: 4.0.5
graylog_sistemas:SECONDARY>

root@server-1:/# mongo --host server-3 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://drjscgerlx043:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“38b460a4-1774-4d53-8f29-bcc7e876105c”) }
MongoDB server version: 4.0.5

root@server-2:/# mongo --host server-1 --port 27017 -u root -p admin20
MongoDB shell version v4.0.5
connecting to: mongodb://server-1:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“2ba6ea25-b0a4-4376-a13c-c23eea4a079d”) }
MongoDB server version: 4.0.5
Server has startup warnings:
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten]
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]

Enable MongoDB’s free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()

graylog_sistemas:PRIMARY>

root@server-2:/# mongo --host server-1 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://drjscgerlx047:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“8fe527df-c653-428c-bbab-4e7f30538ca0”) }
MongoDB server version: 4.0.5
graylog_sistemas:PRIMARY>

root@server-2:/# mongo --host server-3 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://drjscgerlx043:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“2118a1f8-464a-4a0a-a215-493779df1eca”) }
MongoDB server version: 4.0.5

root@server-3:/# mongo --host server-1 --port 27017 -u root -p admin20
MongoDB shell version v4.0.5
connecting to: mongodb://server-1:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“3e316ad7-c30d-4cc4-ae25-0a4d9b7bd9e3”) }
MongoDB server version: 4.0.5
Server has startup warnings:
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten]
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-01-30T15:58:04.501-0200 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2019-01-30T15:58:05.969-0200 I CONTROL [initandlisten]

Enable MongoDB’s free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()

graylog_sistemas:PRIMARY>

root@server-3:/# mongo --host server-1 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://10.216.76.12:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“5935bd9f-4727-4838-933b-c43b829a228f”) }
MongoDB server version: 4.0.5
graylog_sistemas:PRIMARY>

root@server-3:/# mongo --host server-2 --port 27017
MongoDB shell version v4.0.5
connecting to: mongodb://drjscgerlx042:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“68212cca-2c94-4757-845d-a1727a67623e”) }
MongoDB server version: 4.0.5
graylog_sistemas:SECONDARY>

I’m trying to understand why the server-3 didn’t change its status from STARTUP to STARTUP2, so it can get the configuration from the replica set.

root@server-3:/# mongo
MongoDB shell version v4.0.5
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“6e57c0e8-fe73-402d-97ac-3f76ed0a8692”) }
MongoDB server version: 4.0.5

rs.conf()
2019-02-01T11:51:15.565-0200 E QUERY [js] Error: Could not retrieve replica set config: {
“ok” : 0,
“errmsg” : “not authorized on admin to execute command { replSetGetConfig: 1.0, lsid: { id: UUID("6e57c0e8-fe73-402d-97ac-3f76ed0a8692") }, $db: "admin" }”,
“code” : 13,
“codeName” : “Unauthorized”
} :
rs.conf@src/mongo/shell/utils.js:1448:11
@(shell):1:1
rs.status()
{
“ok” : 0,
“errmsg” : “no replset config has been received”,
“code” : 94,
“codeName” : “NotYetInitialized”
}

use admin
switched to db admin
db.auth(“root”, “admin20”);
Error: Authentication failed.

rs.initiate()
{
“ok” : 0,
“errmsg” : “No host described in new configuration 1 for replica set graylog_sistemas maps to this node”,
“code” : 93,
“codeName” : “InvalidReplicaSetConfig”
}

rs.syncFrom(“server-1”)
{
“ok” : 0,
“errmsg” : “not authorized on admin to execute command { replSetSyncFrom: "server-1", lsid: { id: UUID("3af3a940-7be8-46a3-b8b6-232006c71551") }, $db: "admin" }”,
“code” : 13,
“codeName” : “Unauthorized”
}

Another thing that I find very awkward was that information

graylog_sistemas:PRIMARY> rs.printSlaveReplicationInfo()
source: server-2:27017
syncedTo: Fri Feb 01 2019 12:34:23 GMT-0200
0 secs (0 hrs) behind the primary
source: server-3:27017
syncedTo: Wed Dec 31 1969 21:00:00 GMT-0300
1549031663 secs (430286.57 hrs) behind the primary

But the server-3 has the right date. I think that thing is the cause of all the problems. But I couldn’t find the command to sync this.

Any clues?

1 Like
13

Convert it to utc. Maybe it means never synced
I suggest find a mongodb support.

Why mismatch the replica set name ín your connection string and the posted information?

2 Likes
14

I would assume that MongoD simply takes the time from the host OS. This would suggest that not all three hosts are in-sync, meaning you’ll need to take a look at your NTP configuration.

Dec31 1969 is the day before the epoch year, meaning that somehow the clock of one of your hosts is about to roll-back to year 1 of the Unix-clock :smiley: What a wacky situation!

Also, great work on troubleshooting/testing the individual Mongo connections!

1 Like
15

No, check the time zone.

This is at midnight, at UTC, so epoch 0

1 Like
16

Ha!! Nice catch dude! You’re right!

1 Like
17

I think they really never synced, that’s why dates are not the same.
I didn’t understand the last question, sorry.

18

All 3 servers have the right date

  • [user@server-1 ~]$ date
    Mon Feb 4 13:42:52 -02 2019

  • [user@server-2 ~]$ date
    Mon Feb 4 13:42:49 -02 2019

  • [user@server-3 ~]$ date
    Mon Feb 4 13:42:33 -02 2019

Do you think that, if I change server-3 to Arbiter it could sync and solve the problem?

Thanks for all of your help. I’m doing my best.

19

The replica set names are different.

2 Likes
20

Sorry for that.
Because of many problems I had to rebuild the replica set, I made a new one and changed its name.
Everything else are the same.