Estimating disk size


(Jiri Kolb) #1

Hello,
can you please help me with estimation of disk size. I have very simple setup one Fortigate logging via syslog to Graylog ~200 EPS.

Thank you!

Jiri


(Jochen) #2

Hi Jiri,

how big are the log messages on average? How long do you want to keep data until it will be deleted?


(Jiri Kolb) #3

Hi Jochen,
already discussing in this thread:

Thank you for your answer!

Jiri


(Jochen) #4

It looks like you already have all the information you need.

365    * (200 * 60 * 60 * 24)  * $AVERAGE_MESSAGE_SIZE  * (1 + $NUMBER_OF_REPLICAS)                  * 1.5
[days] * [messages per day]    * [average message size] * [number of times the data is stored in ES] * [magic number for the disk space requirements in an ES index]

Message retention
(Jiri Kolb) #5

1,5 is default magic value configured in graylog?

Thank you very much!

Jiri


(Jochen) #6

No, that’s an approximation for how many bytes are required in an Elasticsearch index to store 1 byte of text (due to secondary data structures such as the inverted indices of Lucene).