Master node disappearing intermittently

[Andy.Pieters]

Hi

Using latest Graylog version (4.0.7+c3e766c, codename Noir) with the following configuration:

Linux box 1, Linux box 2, Linux box 3

All three boxes have got elasticsearch running. Elasticsearch is clustered
All three boxes have got mongodb running with replicaset
Graylog is running on all 3 boxes, with box1 being the master

What happens is that everything is fine and dandy and every so often it goes from 3 nodes to 2 nodes and whinges about not having a master node. then a minute later, everything is cool again.

Then a while later it happens again and rectifies itself again.

Very weird.

Any ideas, anyone?

[Andy.Pieters]

This even happens when I’m actually connected to the master node…

I am on the GUI of the master node, and this notification shows intermittently:

[aaronsachs]

Hey there. Reading over this, it makes me think that there is some sort of issue between the master node and MongoDB. Have you looked at the MongoDB logs, or even the Graylog logs to see if that’s the case?

[Andy.Pieters]

Hi

In the mongo logs I see loads of

NETWORK LogicalSessionCacheRefresh

and in the master graylog I see loads of did not find meta

I don’t see any clues that indicate who what why though

(I’m not an expert on graylog, and definitely 98% ignorant on mongo, and 99% ignorant on clustering mongo)

[aaronsachs]

Hmmmm, yeah. Digging doesn’t seem to be giving me a whole lot of results that would fix this when it comes to MongoDB. Could you try running MongoDB 4.2 and see if that fixes it if you’re not already?

[Andy.Pieters]

Hi Aaron

Sadly (…) I am already running 4. : mongodb-org-server-4.2.14-1.el7.x86_64

Further, note that I’ve disabled selinux on each server before installing anything.

I’m using IP addresses in the mongodb_uri, as well as in the config for the mongo replication set.

There’s no firewall blocking traffic on the configured private IP addresses between the nodes, and I’m not using access control on mongo.