Log Retention Strategy

Graylog Central (peer support)
1

All,
I have a question about long term log retention. I’m using and All-In-One Graylog server that has 500GB drive. Graylog is configured for index rotation of 1 Day, Delete index, 90 Max number of indices. I needed to trend some data from last year but unfortunately, I can only go back 90 days.

I’m fully aware of Archiving functionality in Graylog Enterprise. I’m also aware of elasticsearch Index lifecycle management (ILM) policies to automatically manage, but I need to accomplish this in that way. What I was looking at is keeping my index rotation strategy as is, but instead of deleting indices after 90 days I would the close indices out. Maybe I could move the closed indices to a long-term storage (JBOD/NAS). I’m not sure if this works and unsure if I can index them again after moving them. I have also looked at Elasticsearch Snapshot, not sure if it would work in my instance. Looking for some suggestion that I can use that’s not costly. If anyone has option or suggestion that I could use it would be much appreciated.
Thank in advance.

2

You should use curator to automatically backup your indexes and then delete them. You can backup to s3 if you want

3

May be you can use another index set with 365 days retention period for “some data”, which require trending.
Even if you save old indices somewhere else, you will need 4x more space to import them all for year trends.

4

he @gsmith

the question is - will be the 500GB enough for all your ingested data?

5

@jan
The answer to that question will be no, I need at least 2-3 TB of storage. That is if i can compress the old indices into a different volume (i.e. sdb1). Trying to keep Graylog with limit resources, I have a JBOD I can use for storage. I thought… maybe moving the closed Elasticseach Indices and moving somewhere else could be possible?

@ssstonebraker
Ill look into that, Thank you

@Arvo
Basically make a standalone Elasticsearch server?