Graylog stops processing messages after breaching watermark

Graylog Central (peer support)
1

After hitting the low and high watermark we reconfigured elasticsearch.yml as follows:
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 20gb
cluster.routing.allocation.disk.watermark.high: 10gb

I’ve also increased available storage to 100G (was 50) and duly restarted Graylog, and Elasticsearch.

The warnings about hitting the watermark stopped happening but our output buffer and process buffer have filled up to 100% (65K)

Following some advice on the internet, I have backed up and deleted all of my extractors.

Additionally, I shut down Graylog and deleted the index files which some online posts point to being possibly the problem.[1]

However, doing [1] only makes the output buffer start filling up again to 100% which then fills up the process buffer to 100% which then does nothing more useful

[1] Graylog stops processing messages after log flood

2

Check to see if the indexes have been set to read-only. ES tends to do that when it believes disk space is running low.

You can make the indexes writable again using the information here:

2 Likes
3

That’s done it.

Thank you :slight_smile:

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.