Greetings,
I’m a student at the Hochschule Darmstadt in Germany. I’m currently working on a project for my university, where we’re trying to find a suitable Log management tool for our big data cluster.
It will be so helpful if you can provide me with some information.
Equipment:
We have 48 nodes :
28 x Dell PowerEdge C6220
2 Intel Xeon E5-2609 (4 Cores for each)
64 GB RAM
16 x 1 TB SATA 7.2 k
20 x Dell PowerEdge C6320
2 Intel Xeon E5-2620v2 (6 Cores for each)
128 GB RAM
16 x 1 TB SATA 7.2 k
The nodes are connected with a high-bandwidth and low-latency network.
Every node generate for now 500 MB of logs daily, with the total of 24 GB logs daily
The criteria we’re considering are as follows :
- The log management tool should be able to process the generated logs within 10 seconds
Generation ⇒ arrival. In our case this means, from Filebeat until to be ready for search in Graylog.
- Graylog web UI interaction performs within 1 second
We can use 24 nodes in order to scale Graylog and Elasticsearch in order to accelerate the process.
Can Graylog meet these criteria?
Are there any calculations we can do on the speed performance so if the log’s quantity changed we can maintain response tau of 10 seconds?
Your help is very much appreciated.