# Graylog stops processing messages after breaching watermark

**URL:** https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066
**Category:** Graylog Central (peer support)
**Created:** [September 7, 2020, 10:00am UTC](https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066 "2020-09-07T10:00:26Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Andy.Pieters](https://avatars.discourse-cdn.com/v4/letter/a/6f9a4e/32.png) [@Andy.Pieters](https://community.graylog.org/u/Andy.Pieters)
#### Post date: [September 7, 2020, 10:00am UTC](https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066/1 "2020-09-07T10:00:26Z")

</div>

After hitting the low and high watermark we reconfigured elasticsearch.yml as follows:  
cluster.routing.allocation.disk.threshold\_enabled: true  
cluster.routing.allocation.disk.watermark.flood\_stage: 5gb  
cluster.routing.allocation.disk.watermark.low: 20gb  
cluster.routing.allocation.disk.watermark.high: 10gb

I’ve also increased available storage to 100G (was 50) and duly restarted Graylog, and Elasticsearch.

The warnings about hitting the watermark stopped happening but our output buffer and process buffer have filled up to 100% (65K)

Following some advice on the internet, I have backed up and **deleted all of my extractors.**

Additionally, I shut down Graylog and deleted the index files which some online posts point to being possibly the problem.[1]

However, doing [1] only makes the output buffer start filling up again to 100% which then fills up the process buffer to 100% which then does nothing more useful

[1] [Graylog stops processing messages after log flood](https://community.graylog.org/t/graylog-stops-processing-messages-after-log-flood/650/2)

---

<div class="post-metadata">

### Author: ![Ponet](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/ponet/32/2131_2.png) [@Ponet](https://community.graylog.org/u/Ponet)
#### Post date: [September 7, 2020, 11:10am UTC](https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066/2 "2020-09-07T11:10:14Z")

</div>

Check to see if the indexes have been set to read-only. ES tends to do that when it believes disk space is running low.

You can make the indexes writable again using the information here:

> <https://stackoverflow.com/questions/34911181/how-to-undo-setting-elasticsearch-index-to-readonly/50954055#50954055>

---

<div class="post-metadata">

### Author: ![Andy.Pieters](https://avatars.discourse-cdn.com/v4/letter/a/6f9a4e/32.png) [@Andy.Pieters](https://community.graylog.org/u/Andy.Pieters)
#### Post date: [September 8, 2020, 7:29am UTC](https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066/3 "2020-09-08T07:29:11Z")

</div>

That’s done it.

Thank you 🙂

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)
#### Post date: [September 22, 2020, 7:29am UTC](https://community.graylog.org/t/graylog-stops-processing-messages-after-breaching-watermark/17066/4 "2020-09-22T07:29:14Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
