Ive set up a production cluster consisting of a elasticsearch cluster with 3 nodes and 3 graylog servers running mongdb with replicaset.
I set the shard number to 3 (One per node)
And the replica to 1.
Now i’m planning on how I’m going to organize my logs in it. I need to retain 6 months worth of logs for every devices sending logs to it for “hot access” and I was thinking of having separated indices for every device for better archiving purposes (I need to store logs for 7 years). And then it comes down to:
If I set my retention strategy to 1D and retain 180 days, in 6 months I’m gonna have 540 shards and 540 replica shards per device! So, 1080 shards per device seems a LOT! Is it a problem?
When I run a search, is it possible to search all indices with the same prefix? I know I can search a specific index but it seems I can’t use wildcards for this field.
I read a lot of documentation but I’d like some thoughts from experienced people about this.