yes!
You create new shards every time your index rotates. If you rotate every day you will create new shards every day. But you will also delete the oldest ones → the number of shards will stay constant.
My default setting is to set the rotation on P1D → one day. If I ingest more than 30GB of data into that index per day, I will set the number of shards to two, if I have more than 50 to three and so on. The daily amount is the relevant amount, as I rotate every day.
Total used disk space is not so relevant here, as it depends how many indices I will keep. If I want to have 90 days of data it will be more than just for 10 days.
Your index set A with 10GB in total in 30 days is perfectly fine with only one shard per day.
Your index set B with 200GB in four days is a bit strange. I’d go for a daily rotation with two shards.
Two important things for the performance:
- per 20 shards you should allocate 1GB of Heap for your Elastic/Opensearch
- for each GB heap of Elastic/Opensearch you should have one GB of RAM for buffers by the OS.
Example:
16GB Ram for your Elastic/Opensearch-Machine, no Graylog or Mongo on that machine.
8GB Heap for Elastic/Opensearch, 8 for OS and Caches
8GB Heap * 20 shards = 160 shards