Good morning community,
i am a newnubbie to Graylog and ES and i would like a design suggestion from you.
We need to index 5 years old logs taken from a SIEM from a subset of applications, we have one server (system spec should be defined) and an attached NFS NAS storage of 5 TB. The logs size to be ‘uploaded’ once a day (we though to retrieve and upload the data via API) each day are few, approx 400 MB per day; so basically 712GB per 5 years. I have the following questions, i hope you can help me understand better and give me some advice:
1- It is feasable to use the 5TB NFS as a index data store nonetheless the few data required to be indexed? As far as i understood NFS datastore are a nono in ES.
2- All the log data from 5 years back should be searchable and indexed; an index of almost 1TB is feasable? Or it is adviced to define an index for each log source and/or for each day/week/month? What are your suggestion about index and shards to be defined?
3- With the data required to be uploaded and indexed, what server (1 node) specs do you suggest? 8cpu and 32GB RAM?
4- If the NFS datastore is not an option how can i use it? Store Index snapshots maybe?
I hope my questions are clear enough and that you can help me on my doubts.
Thanks,