I am looking at creating an instance with about 10,000 messages per second.
What setup do people run an instance of this size on, would 32gb ram and 8 cores work?
I know the above is quite vague, but from experience what does everyone suggest?
Cheers,
George
you should give Elasticsearch enough ressources - those amount of messages should work with a single Graylog Node, but should have multiple Elasticsearch Hosts in a cluster.
We are going into a full production service and have never worked with clustering so don’t want to be experimenting.
On the other hand I understand clustering should be used in production as it’s more reliable and efficient.
For now I think we will have to just run with a single node and try to build a test cluster to see how it all works.
Cheers,
G
6 posts were merged into an existing topic: Message processing from kafka
Just a thought:
If you have your single server Graylog, setup next to it a elasticsearch cluster like https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04
Then ajust the /etc/elasticsearch/elasticsearch.yml of the graylog server so it reflects being master of the cluster.
As is mentioned in the above howto, multiple nics needed, in single lan setup security is an issue.
If you wish to avoid clustering you may need to go bearbone hardware setup to obtain enough cpu and IOPS
A good raid with ssd for hot storage (and maybe sata for warm/cold storage )or storage system, together with enough RAM recommended…
Virtualisation is great, but sometimes direct hw better (/cheaper). 
Cheers…
Arno
I’m seriously considering clustering now.
Could I have 1 server (64gb ram 8 cores) with GL and ES on then 2 server with just ES on the create the cluster or should GL be separated from the ES cluster?
When messages are being processed in the Graylog node section of the web interface, is that showing Graylog receiving messages and send them onto ES?
I still feel like I have a lot to understand so please bear with me.
I would give GL one Server and ES two servers - that they did not share ressources.
What specs would you recommend for those three servers?
I was thinking
GL: 32gb RAM (16 for GL) 8 cores
ES x 2: 64gb RAM (32 for ES) 8 cores
Cheers,
G
That sounds like a plan!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
