Confused on why I'm not storing as much as I expected

(Jason Haar) #1

Hi there

I have graylog configured as follows

rotation_strategy = size
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_size_per_index = 10737418240
elasticsearch_max_number_of_indices = 2000
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 2

From this I work out that I should be using about 10Gx2000 = 20TB (I know, x1.5 to take into account ES indexing/etc) before the deletion of old indexes kicks in, and on top of that the replicas = 2 should mean this is using 40TB (or is it 60TB?) of diskspace aggregated on the cluster members…

But on this 5-node ES cluster, I’m only seeing 5x1.4TB of diskspace in use. It’s ‘steady state’ - I’m seeing the deletion rotation policy triggering in the logs - so obviously I’ve screwed something up. Any ideas?

Thanks

(Jochen) #2

Which version of Graylog are you using?

Starting with Graylog 2.2.0, the configuration of index settings has been moved into index sets (and thus into the database): http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration

(Jason Haar) #3

I’m on the latest via yum under CentOS-7. ie

graylog-server-2.3.0-7.noarch
elasticsearch-2.4.6-1.noarch

But thanks - you’ve answered my question. I was unaware of this change. I used to have elasticsearch_max_number_of_indices set to 800 - but increased it in the config - but obviously that doesn’t work any more. But looking in the ‘System->Indices’ area, I see it’s still set to 800. Now I know what to do :slight_smile:

Thanks!

Jason

(system) closed #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.