I’m having quite a bit of problems. While trying to fix {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}, I tried the steps at Restart collecting messages but now I’m just getting blocked by: [FORBIDDEN/7/index read (api)]; when I try to search for messages.
he @mmahacek
you reach the high/low watermark on your Elasticsearch server(s).
The following settings allow to control when Elasticsearch will first stop allocating new shards, and when it starts to relocate shards or when it sets the Cluster into read-only state because to many messages are coming in.
Setting: cluster.routing.allocation.disk.watermark.low
Controls the low watermark for disk usage. It defaults to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent ES from allocating shards if less than the configured amount of space is available.
Setting: cluster.routing.allocation.disk.watermark.high
Controls the high watermark. It defaults to 90%, meaning ES will attempt to relocate shards to another node if the node disk usage rises above 90%. It can also be set to an absolute byte value (similar to the low watermark) to relocate shards once less than the configured amount of space is available on the node.
Setting: cluster.routing.allocation.disk.watermark.flood_stage
Controls the flood stage watermark. It defaults to 95%, meaning that Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node that has at least one disk exceeding the flood stage. This is a last resort to prevent nodes from running out of disk space. The index block must be released manually once there is enough disk space available to allow indexing operations to continue.
On a 5TB Disk the following settings are safe:
/etc/elasticsearch/elasticsearch.yml
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb
That’s what I thought, but my Elastic cluster is showing around 50% disk usage on the data partition.
I guess that your Index retention and rotation has made some space available again.
If you have metrics of your system you might want to check these for that.
Indices had rolled over, but were stuck a zero bytes. It was weird that I couldn’t even query the raw data directly in Elastic going back several days. Regardless, it is working now, after purging some of my indices.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.