Datanode refuse to start after upgrade

On outofthe install on docker environment.

“disk usage exceeded flood-stage watermark, index has read-only-allow-delete block” force datanode to shutdown.


Impossible to connect on 9200 (server shutdown automatically).

System seems to by clean.

Filesystem                     1K-blocks     Used Available Use% Mounted on
overlay                         25692476 16901156   7558604  70% /
tmpfs                              65536        0     65536   0% /dev
shm                                65536        0     65536   0% /dev/shm
/dev/mapper/ctf1--vg-docker     25692476 16901156   7558604  70% /etc/hosts
/dev/mapper/ctf1--vg-challenge   8154588   540548   7178228   8% /var/lib/graylog-datanode
tmpfs                           10240128        0  10240128   0% /proc/acpi
tmpfs                           10240128        0  10240128   0% /sys/firmware

Btw it work with the last 5.8.x version.
(I recreate infra with “cluster.routing.allocation.disk.threshold_enabled: false”)

Is it a good way to deal with this issue ?

Thanks for you help.


datanode-1 | 2024-07-07T21:20:41.499Z INFO [OpensearchProcessImpl] [2024-07-07T21:20:41,498][WARN ][o.o.c.r.a.DiskThresholdMonitor] [datanode] flood stage disk watermark [95%] exceeded on [TOHdPf2JShyepL42R9sZ_w][datanode][/var/lib/graylog-datanode/opensearch/data/nodes/0] free: 824kb[0%], all indices on this node will be marked read-only

df /var/lib/graylog-datanode/opensearch/data/nodes/0
Filesystem                     1K-blocks   Used Available Use% Mounted on
/dev/mapper/ctf1--vg-challenge   8154588 529260   7189516   7% /var/lib/graylog-datanode

After node in read-only — datanode refuses to start with
[ProcessWatchdog] Process watchdog terminated after too many restart attempts

are you ingesting data into Graylog (if yes, at what rate?) or is this happening without ingestion?