Graylog retention has any affect on shards?

[groovyghost (Rahul)]

I’m new to graylog.
I need to create a index set for 2 years.really confused how to set retention time with max indices.if i keep max number of indices high with 4 shards will it affect performance ?pls help

Graylog 2.4 in centos 7

[gsmith]

Hello && Welcome

@groovyghost

I need to create a index set for 2 years.

Now that’s a long time to hold data.

@groovyghost

eally confused how to set retention time with max indices.

If your index rotation is set for a day then you would set your retention for 730

Example:

@groovyghost

if i keep max number of indices high with 4 shards will it affect performance

Yes it will, but it also depends on your searches specially if you have some 730 indices then multiply that by 4 you get 2920 shards. Elasticsearch will need resources plus the volume will need to be calculated for holding that many messages. Again this will depend on how many messages you are ingesting, for how long and what your search criteria is for dashboard and saved searches.

You may want to start read here for a better understand what you might need
.
https://docs.graylog.org/docs/getting-started

Also check out this post

Message retention

Hello, We have very simple setup just one Fortigate firewall with estimation of 200 EPS that is 288000 events per day so maximum size is approximately 100MB per day? Would be great if you can provide values from your environment. Regarding indexes deletion that is pretty straitforward that index is deleted from db. I read the links but if you can explain to me in more practical way what is close index. Is information in close index available for search? Indexes that are closed are not stored in…

Hope that helps

[groovyghost (Rahul)]

Thanks for the reply we have settled on 1week per index with max of 110 weeks (2plus years)