after a month I wanted to access the Graylog web interface again. However, it is no longer accessible.
With the command “sudo tail -f /var/log/graylog-server/server.log” I get the following output: ERROR [VersionProbe] Unable to retrieve version from indexer node: Failed to connect to 127.0.1.1:9200. - Connection refused.
INFO [VersionProbe] Indexer is not available. Retry #196
with the command: dpkg -l | grep -E “.(elasticsearch|graylog|mongo).”
I noticed that elasticsearch is no longer available.
Now to my question: why is elasticsearch suddenly no longer available and how do I restore it without losing data?
OS Information: Ubuntu 24.04.1 LTS
Package Version: 6.1.3-1
So anywhere you see it mention elastic etc in the logs that will just mean datanode for you. So i would give the datanode service a restart, and then check the datanode logs for any issues.
I found the following in the logs:
[OpensearchProcessImpl] org.opensearch.cluster.block.ClusterBlockException: index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
How can I delete the read-only-allow-delete block?
I have already tried “curl -X GET ‘http://127.0.0.1:9200/_cluster/health?pretty’, but I get a ”curl: (7) Failed to connect to 127.0.0.1 port 9200 after 0 ms: Could not connect to the server"
You basically ran out of disk space (Opensearch doesn’t like getting past 70-80%) You will either need to expand the drives, or you could have it delete some of the older indices.