Graylog System requirement and indices configuration


(Irfan00001) #1

hello everyone, i have some basic queries which i need to clear for graylog 3.x
first query is what will be the hardware requirement in the below two scenarios

  1. for UAT server we are getting 2000 requests per hour
    2.for production server we are getting 20000 request per hour
    so in both the above scanarios what will be the hardware specification for the greylog and we are also installing mongodb and elastic search on the same server.

My second query is how does this indices works. i am finding very difficult to understand the indices concept. my requiremnt is to store the logs for atleast minimum one year,then it should automatically get deleted. so how can we configure index sets to store the logs for one year.

Please help me out on this.

Thanks,
Irfan


#2

Here is the graylog sizeing calculator.
https://docs.google.com/viewer?a=v&pid=forums&srcid=MTMyNzU4MDI3MTY0NTIwNzM3MDcBMTc1MTY2ODg0OTMxNzA5MTc1OTYBbHpVZEN5SDNBUUFKATAuMQEBdjI

You maybe think for indices. You have to set Time based rotation, New index after 1 day, delete after 365 days.
Indices contain shards. There is no limit about that, but the suggestion to keep the size of shards between 20 and 40 GB.
You also have to calculate with elastic memory. If you want to search in the one year logs, you need about 2% memory for elasticsearch of the RAW log data. Eg. If you get 480k log/day, 1kb avarage size, 480mb/day -> 175Gb/year 3,5 GB elasticsearch heap.


(Irfan00001) #3

Hi Mack,

Thank you for the help.

But I am not able to find the Graylog Sizing Calculator.
Please help with the details.

Thanks,
Irfan


#4

Sorry, I checked, and it was a wrong link.
I updated my post (to remove the wrong data).
There is the good link.


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.