Message retention

Disk Usage

Well, as @jochen already said in your other topic:

I looked at our Fortigate input and the messages were about 380 - 450 bytes in size. (So lets assume 415 bytes per message for the calculation (Keep in mind that you probably need some more if you parse your message into many fields).

  365   * (200 * 60 * 60 * 24) *        415 byte        *           (1 + 0)          *       1.5      = 3926232000000  bytes / 1099511627776 =  3,5708 TiB ( 3,9261 TB) per year // Just one ElasticSearch Node
  365   * (200 * 60 * 60 * 24) *        415 byte        *           (1 + 1)          *       1.5      = 7852464000000  bytes / 1099511627776 =  7,1417 TiB ( 7,8523 TB) per year // Standard 2-Node-Cluster
  365   * (200 * 60 * 60 * 24) *        415 byte        *           (1 + 2)          *       1.5      = 11778696000000 bytes / 1099511627776 = 10,7126 TiB (11,7786 TB) per year // 3-Node-Cluster
  365   * (200 * 60 * 60 * 24) *        415 byte        *           (1 + 3)          *       1.5      = 15704928000000 bytes / 1099511627776 = 14,2835 TiB (15,7048 TB) per year // And so on ...
 [days] *  [messages per day]  * [average message size] * [primary + replica shards] * [magic number] = result

(1099511627776 is the factor for bytes to Tebibytes, in brackets are actual Terabytes (calculated with 1000 instead of 1024)) As you see, for more replicas it just scales by the number of replicas, this should be obvious.

Breaking this down, you would need 3,5708 TiB (3,9261 TB) / 365 = 10,0178 GiB (10,7564 GB) per day.

Closed Indexes

In short, closed Elasticsearch indexes just reside on disk. They are not accessible for search, since they won’t be loaded into memory for search operations. See it as a step in between active data you use and inactive data you archived on another system. It is not accessible like active data, but can be re enabled with an API-call instead of having to copy data from an archive system.

Greetings - Phil

PS: I guess you miscalculated. 200 EPS calculated to a day is 17280000, not 288000. You calculated 200 * 60 * 24 which means 24 minutes. :slight_smile:

1 Like