Multi-site considerations

Hello all,

Let’s assume an installation of four graylog nodes on two datacenters, two on each. One of the datacenters is considered primary. The four graylogs are supported by a single five-node mongodb cluster and a five-node opensearch cluster; in both implementations, the 5th node is an arbiter that sits on the primary datacenter.

Now, let’s assume the following two scenarios:
S1 - primary datacenter goes completely offline
S2 - the connection between the two datacenters gets severed for X minutes

I understand that in both scenarios, mongodb will become readonly on the secondary site, and graylog stops working. What happens to log collection in that case?

Let’s assume that I started an arbiter in the secondary site and therefore graylog resumes. When we recover from S1, primary datacenter would catch up and all is good. (correct me if wrong).

What happens however in S2, where the primary datacenter had a working mongodb cluster (thanks to the arbiter) and never stopped collecting? If I have started an arbiter in the secondary site, then I would have a diverged replicaset, and the cluster would reject some of the nodes, if not all. (A mongodb recovery is required then)

I understand that this may look like a mongodb problem and not related to Graylog, but my question remains: what is the best way to implement a fault-tolerant graylog cluster?

Can I run a single graylog cluster with two mongodb clusters? (one for each site)?

(I’m obviously not worried about the opensearch cluster going offline temporarily as graylog can work with it offline for a few hours depending on the load.)

Thanks in advance and apologies for the long post.

  • AKG

Hey @akg

To be honest, If I had two clusters, in two different data centers, they would consist of ES/OS, Graylog and MongoDb. Going off your statement, I would forward my logs from DC-2 to DC-1. MongoDb just holds metadata and my main worrry would be in ES/OS which holds all the data. Just an idea.

Hello @gsmith, thanks for your reply!

This kinda answers my question. As far as you (or anyone) know, can I have a single ES/OS cluster for two graylog clusters, feeding into the same index? I guess the answer is yes but a confirmation would be nice.

No not really, under perfect conditions maybe, but it is definitly not built to support that, so i would not go near it with a 10 foot pole. You could have one elastic cluster hosting the database of both, but they would not be related in any way.

1 Like

Hey @akg

I agree with @Joel_Duffield

For example:
I have 6 nodes in Germany, 6 nodes in UK and 6 nodes here In Iowa. Each of the 6 nodes consist of 3 OpenSearch nodes and 3 Graylog/MongoDb nodes. UK nodes forwards Log to Iowa, Germany cluster Forwards logs to Iowa. In case of Internet interruption or issues each DMZ is contained till the connect comes back. These are all runned through a VLAN and some other security stuff.

On the cluster in Iowa, I have a index set called UK and another called Germany from there I create alerts and widgets to those geographical locations. It all breaks down on your environment and the ability to expand if need be.
Hope that helps.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.