# Data retention for the whole Year

**URL:** <https://community.graylog.org/t/data-retention-for-the-whole-year/30397>\
**Category:** Graylog Central (peer support)\
**Created:** [October 19, 2023, 8:31pm UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397 "2023-10-19T20:31:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![henryd99](https://avatars.discourse-cdn.com/v4/letter/h/258eb7/32.png) [@henryd99](https://community.graylog.org/u/henryd99)\
**Post date:** [October 19, 2023, 8:31pm UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/1 "2023-10-19T20:31:48Z")

</div>

Hi all,  
I have a Input from our CGNAT and need to store data retention for 1 year for IP usage.  
How can I set this, and what is the most efficient configuration for Index Rotation Configuration and Index Retention Configuration.  
Im not sure what value should be used.  
Thank you!

---

<div class="post-metadata">

**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\
**Post date:** [October 19, 2023, 8:43pm UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/2 "2023-10-19T20:43:11Z")

</div>

How much data are you going to be receiving every day, both the number of messages and the total size of those messages?

---

<div class="post-metadata">

**Author:** ![henryd99](https://avatars.discourse-cdn.com/v4/letter/h/258eb7/32.png) [@henryd99](https://community.graylog.org/u/henryd99)\
**Post date:** [October 19, 2023, 8:55pm UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/3 "2023-10-19T20:55:13Z")

</div>

It would be 40,000 messages per day, but Im not sure how to see the size of each one. Where can I find that info? Sorry im new to graylog.

---

<div class="post-metadata">

**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\
**Post date:** [October 20, 2023, 1:02am UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/4 "2023-10-20T01:02:50Z")

</div>

If you are already ingesting the messages you can just post a picture of the graph from the system\>overview page, the will show volume for the last 30 days.

---

<div class="post-metadata">

**Author:** ![henryd99](https://avatars.discourse-cdn.com/v4/letter/h/258eb7/32.png) [@henryd99](https://community.graylog.org/u/henryd99)\
**Post date:** [October 26, 2023, 10:02am UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/5 "2023-10-26T10:02:02Z")

</div>

As its only recent I can only get the daily which is 3800M bytes  
The image below is for 6 customers at the moment. 3800M would be for around 200 customers per day.  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/8/8169d5535c25d6d79a0410f6ba70061cf09440ae.png)

---

<div class="post-metadata">

**Author:** ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)\
**Post date:** [October 26, 2023, 11:58am UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/6 "2023-10-26T11:58:06Z")

</div>

That seems managable for one year 😃

1. put your messages all in a unique stream
2. create an index set with the weekly rotation (P1W), 53 Max number of indices, rotation strategy based on Index Time
3. configure your stream to use the indexset

Make sure to maybe split your opensearch-database above two machines, and to configure Index replicas to 1. One of your machines might die, and you will not lose data.

If your amount of data is growing it will be more of a challenge, but ~100MB/day should be fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [November 9, 2023, 11:59am UTC](https://community.graylog.org/t/data-retention-for-the-whole-year/30397/7 "2023-11-09T11:59:03Z")

</div>

Closed on November 9, 2023, 11:59am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
