OpenSearch failed to start after reaching high watermark

Im using Graylog 6.0.0-rc.5-1 with the official docker-compose from github:

The instance reached 80% disk usage and OpenSearch changed to read-only as expected. I extended the disk with 100G more and after a restart/reboot the graylog-datanode container still in read-only mode:

2024-05-07T07:22:49.815Z WARN  [OpensearchProcessImpl] ClusterBlockException[index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

I checked and there should be enough space now:

docker exec -it f43ef4e460da bash
root@datanode:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         186G   66G  112G  38% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/vda2       186G   66G  112G  38% /etc/hosts
tmpfs           7.8G     0  7.8G   0% /proc/acpi
tmpfs           7.8G     0  7.8G   0% /sys/firmware

I was trying to resolve the issue with simply allow write on OpenSearch but I found out that I cannot run any API call on OpenSearch:

curl -k -u admin:xxxxxxxxxxxxxxxxxxx https://127.0.0.1:9200
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9200

Im not sure how i supposed to interact with the opensearch inside the graylog-datanode container, and I cannot find any information about what is the default user/ what password should i use with this graylog/graylog-datanode image.

Most of the documentation I found using the “regular” elasticsearch/opensearch image, not the graylog/graylog-datanode.

So maybe someone with more experience with graylog-datanode could help me…

Hey @csubee

Instead of HTTPS have you tried HTTP for 127.0.0.1:9200?