Need Advice on Scaling Graylog Open 6.x – Hitting Performance and Storage Limits

[Golobolus (Petrov Diabolique)]

Hi Everyone,

We’re currently running Graylog Open 6.x, and over time our log ingestion has grown significantly — from around 10GB/month to roughly 100–200GB/month today.

Our current setup is a single VM running all Graylog components (Graylog server, MongoDB, and OpenSearch). As expected, this setup is now struggling — we’re seeing data pull timeouts, unresponsive VM instances, and missing logs quite often.

Before we consider moving to the Graylog Enterprise License, I’d like to explore if we can improve or re-architect our current setup using the Open version.

We’re planning to test a new setup with:

• 2 Graylog Open instances

• 4-node DataNode cluster

However, from what I understand, clustering multiple Graylog application nodes is only supported under the Enterprise version.

So my main questions are:

1. Would this kind of setup (2 Graylog Open instances + 4 DataNodes) actually work in practice?

2. If not, what’s the best way to scale or distribute the load using Graylog Open?

3. Are there any proven setups or AWS best practices that can help stabilize performance before going Enterprise?

For context:

• Hosted on AWS

• Logs stored on EFS, currently under one EC2 instance

Any advice, references, or insights from those who’ve scaled Graylog Open would be greatly appreciated.

Thanks in advance!

[Joel_Duffield]

This video is a little older but it mostly still applies, and will give you an idea of scale etc. https://www.youtube.com/watch?v=agdLrDw9JaE
You can cluster in open, that is not an enterprise only feature (would be interested to know where you saw that), you will most likely have to put your own load balancer in front of the Graylog nodes though for web and message traffic.

The limitation you will have in open is that you won’t see the performance metrics of your datanode cluster, where as in enterprise you will see that in the UI among lots of other things.