What should be the ES configuration if we store 8TB of data per day and want to retain only 10 days of data in ES?
Number of ES master nodes?
Resource for ES master nodes?
Number of ES coordinating nodes?
Resource for coordinating nodes?
Number of ES data nodes?
Resource for ES datanodes?
Max no of indices?
Field type refresh interval time?
Index Rotation period?
Replicas: 0 (I don’t want replicas)
Number of Shards per index?
A quick response will be appreciated.
Thanks,
As always it depends… 
I know of one company (bol.com) that was working with almost the same amount of data in ES (11TB), maybe their presentation once done at a elastic presentation can be found on-line
They had a Redis server in front of their ELK cluster.
That consisted out of at least one coordinating node and around 10 ES data nodes.
I once figured out some statistics to make calculations and optimize our system.
• 3, 6 of 9 primary shards per index, depending on speed wanted and number of in our case 3 data nodes.
• Shard size between 10 tot 25GB for fast search, if bigger search becomes slower
• 20 shards per GB heap space (12GB heap = 240 shards/node =750 shards that default to 1000 on a cluster per node)
• Memory:Disk == 1:16 on average
I see you do not want replicas, but they can speed up search by 50%.
A coordinating only node is preferred above 5 or 6 elastic nodes.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
