You can find the sharding recommendations here.
As a rule of thumb, you should aim for 10GB+ shards. So how many of them you set will depend on how much data you are collecting, and how long you are storing it for.
Can you post a screenshot of your indicies setup, including how many shards they have, rotation strategy etc.
yes
batch size is 500
21 shards
index rotate every 1 hr
PT1H
21 shards per index (so 21 new shards every hour when it rotates), or 21 shards total?
21 shards per index
yes 21 new shards every hour
So first off, if you upgrade to graylog 5.1 you can switch over to time size optimization of index rotation and graylog will look after rotating then at the right time. Also you want shards ro be a multiple of your nodes, but I would just start with the same number of shards as nodes, see OpenSearch Shard Size: Choosing the Correct Number of Shards
1 Like
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.