Hi guys. This is my forst post here on community; I’m very new to Graylog, so I could make questions abvious for you but not form me. Sorry about that!
I have this scenario. For a customer, we are implementing an architecture like this:
Data sources → Log collector → Other systems
Based on architectural design, Graylog is one of chosen Log collector software. Now, Data sources will be many, but I have an estimation of ho much EPS we’ll have: the extimated spike is about 2000.
Based on this, which hardware requirements will grant us a stable system able to work with this data size?
As unhelpful as it may sound, the short answer is “it depends”. Meaning there are a some variable that go into determining how much CPU/RAM you need for a given log ingestion volume. It also helps to talk about this in daily ingest volume since EPS (events per second) could mean many different things, such as average message size is 400 bytes vs 1600 bytes.
With that said, napkin math of 2,000 EPS at an average of 800 bytes per message is roughly 138 GB/day.
Other variables that can contribute to sizing:
EPS and SIZE of messages
processing rules
how long messages are stored in OpenSearch (what we call the indexer, basically the database)
For graylog I would recommend at least 3 graylog server nodes with at least 8 vCPUs and 8GB of ram.Your mileage may vary so some tuning (such as increasing CPU count) may be required once implemented and you can see how your environment behaves and what the resource utilization looks like. A load balancer can be configured to in front of the graylog nodes.
Mongo (either a single node or in a replica set) can be installed on the graylog nodes above and has minimal requirements.
For OpenSearch, at a minimum 2 OpenSearch nodes with 32GB of RAM (1/2 of which is assigned to heap, or 16GB on each node) and 8 vCPUs.
Graylog does scale horizontally very easily, meaning you can add nodes to both the graylog cluster and the opensearch clusters respectively to increase capacity. You’ll generally get better efficiencies with slightly more nodes with less resources vs. vertically scaling up only a couple of servers with maxed out stats.
