I am running 3 Elasticsearch Nodes in a Cluster with 2 Graylog Servers.
We have different Webservers and individual Logs which we want to put on the first Graylog Node.
With the second Graylog Node we want to handle our Domain Controller Logs, which is about 2000msg/s.
We have to store this big amount of data for half a year.
If I activate the Input of DC Logs on my second node, the elasticsearch Cluster perfomance is very heavy. (4CPU, 16GB RAM, 500GB SSD Storage each)
However, I am looking for a solution to kinda archivate the DC Logs. We do not need to select and filter them every day and they should also not influence the other “Hot” Logs on the first node.
Is there any Chance to build such an Environment? Do I have to use the Graylog Enterprise Version? If yes, just on the Second node? Can I directly store the logs on a cheap disk space? Will they then also consume our Elasticsearch Cluster Performance?
Any other ideas to implement this?
I really appreciate! Thank you in advance.