On this topic: Multi-site considerations - #2 by gsmith
These comments recommended the below, maybe it would be good in my situation?
From user gsmith
To be honest, If I had two clusters, in two different data centers, they would consist of ES/OS, Graylog and MongoDb. Going off your statement, I would forward my logs from DC-2 to DC-1. MongoDb just holds metadata and my main worrry would be in ES/OS which holds all the data. Just an idea.
For example:
I have 6 nodes in Germany, 6 nodes in UK and 6 nodes here In Iowa. Each of the 6 nodes consist of 3 OpenSearch nodes and 3 Graylog/MongoDb nodes. UK nodes forwards Log to Iowa, Germany cluster Forwards logs to Iowa. In case of Internet interruption or issues each DMZ is contained till the connect comes back. These are all runned through a VLAN and some other security stuff.
On the cluster in Iowa, I have a index set called UK and another called Germany from there I create alerts and widgets to those geographical locations. It all breaks down on your environment and the ability to expand if need be.
Hope that helps.