Estimating disk size

[colbik (Jiri Kolb)]

Hello,
can you please help me with estimation of disk size. I have very simple setup one Fortigate logging via syslog to Graylog ~200 EPS.

Thank you!

Jiri

[jochen]

Hi Jiri,

how big are the log messages on average? How long do you want to keep data until it will be deleted?

[colbik (Jiri Kolb)]

Hi Jochen,
already discussing in this thread:

Message retention

Hello, I would like to store logs for one year then delete. Are following values correct? According to ISO 8601 one day is P1D is so rotation period set to:P1D Select retention strategy set to:365 Select retention strategy set: delete Can you please also explain to me the difference between close and deletion of retention strategy? Thank you! Jiri

Thank you for your answer!

Jiri

[jochen]

It looks like you already have all the information you need.

365    * (200 * 60 * 60 * 24)  * $AVERAGE_MESSAGE_SIZE  * (1 + $NUMBER_OF_REPLICAS)                  * 1.5
[days] * [messages per day]    * [average message size] * [number of times the data is stored in ES] * [magic number for the disk space requirements in an ES index]

[colbik (Jiri Kolb)]

1,5 is default magic value configured in graylog?

Thank you very much!

Jiri

[jochen]

@colbik

1,5 is default magic value configured in graylog?

No, that’s an approximation for how many bytes are required in an Elasticsearch index to store 1 byte of text (due to secondary data structures such as the inverted indices of Lucene).