Im using Graylog 6.0.0-rc.5-1 with the official docker-compose from github:
The instance reached 80% disk usage and OpenSearch changed to read-only as expected. I extended the disk with 100G more and after a restart/reboot the graylog-datanode container still in read-only mode:
2024-05-07T07:22:49.815Z WARN [OpensearchProcessImpl] ClusterBlockException[index [.opensearch-observability] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
I checked and there should be enough space now:
docker exec -it f43ef4e460da bash
root@datanode:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 186G 66G 112G 38% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/dev/vda2 186G 66G 112G 38% /etc/hosts
tmpfs 7.8G 0 7.8G 0% /proc/acpi
tmpfs 7.8G 0 7.8G 0% /sys/firmware
I was trying to resolve the issue with simply allow write on OpenSearch but I found out that I cannot run any API call on OpenSearch:
curl -k -u admin:xxxxxxxxxxxxxxxxxxx https://127.0.0.1:9200
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9200
Im not sure how i supposed to interact with the opensearch inside the graylog-datanode container, and I cannot find any information about what is the default user/ what password should i use with this graylog/graylog-datanode image.
Most of the documentation I found using the “regular” elasticsearch/opensearch image, not the graylog/graylog-datanode.
So maybe someone with more experience with graylog-datanode could help me…