Unable to log into web UI and [TOO_MANY_REQUESTS/12/disk usage exceeded

Hi, my symptoms are that I can’t log into the WEB UI, error message: There was an error fetching a resource: Service unavailable, Authentication service unavailable.

I have two servers, Redhat 8.8, one running Graylog 5.0.9, the other opensearch (or elasticsearch) 5.3.7

When I try to log into the web UI I get:
Error - the server returned: 503 - There was an error fetching a resource: Service Unavailable. Additional information: Authentication service unavailable

I see this in server.log:
index [syslog_0], id [543bc63a-c930-11ed-b4be-005056a5f381], message [OpenSearchException[OpenSearch exception [type=cluster_block_exception, reason=index [syslog_0] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]]

If someone could put me on the path to fix this I would really appreciate it. Each time my server team patches the OS / App they have had to increase disk space, this condition started after patching.

Greetings. This mean the disk space usage where elasticsearch/opensearch is running has exceeded the high watermark threshold, which is 90% i believe. See Fix common cluster issues | Elasticsearch Guide [7.17] | Elastic

You’ll need to free up space. My guess is the retention policies in graylog are allowing more data to be stored than there is available on the disk.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.