Datanode refuse to start after upgrade

Graylog Central (peer support)
1

Hi,
On outofthe install on docker environment.

“disk usage exceeded flood-stage watermark, index has read-only-allow-delete block” force datanode to shutdown.

DATANODE_IMAGE=graylog/graylog-datanode:6.0.4
GRAYLOG_IMAGE=graylog/graylog:6.0.4

Impossible to connect on 9200 (server shutdown automatically).

System seems to by clean.

Filesystem                     1K-blocks     Used Available Use% Mounted on
overlay                         25692476 16901156   7558604  70% /
tmpfs                              65536        0     65536   0% /dev
shm                                65536        0     65536   0% /dev/shm
/dev/mapper/ctf1--vg-docker     25692476 16901156   7558604  70% /etc/hosts
/dev/mapper/ctf1--vg-challenge   8154588   540548   7178228   8% /var/lib/graylog-datanode
tmpfs                           10240128        0  10240128   0% /proc/acpi
tmpfs                           10240128        0  10240128   0% /sys/firmware

Btw it work with the last 5.8.x version.
(I recreate infra with “cluster.routing.allocation.disk.threshold_enabled: false”)

Is it a good way to deal with this issue ?

Thanks for you help.
Whoo

2

opps

datanode-1 | 2024-07-07T21:20:41.499Z INFO [OpensearchProcessImpl] [2024-07-07T21:20:41,498][WARN ][o.o.c.r.a.DiskThresholdMonitor] [datanode] flood stage disk watermark [95%] exceeded on [TOHdPf2JShyepL42R9sZ_w][datanode][/var/lib/graylog-datanode/opensearch/data/nodes/0] free: 824kb[0%], all indices on this node will be marked read-only

df /var/lib/graylog-datanode/opensearch/data/nodes/0
Filesystem                     1K-blocks   Used Available Use% Mounted on
/dev/mapper/ctf1--vg-challenge   8154588 529260   7189516   7% /var/lib/graylog-datanode

After node in read-only — datanode refuses to start with
[ProcessWatchdog] Process watchdog terminated after too many restart attempts

3

@whoo
are you ingesting data into Graylog (if yes, at what rate?) or is this happening without ingestion?

4

Hi,
It’s happen with a normal rate. ~10MB/day.
No issue with 5.2.9.

5

Do I need to use opensearch instead of graylog/graylog-datanode:6.0.4 ?

6

Found a working configuration

  • opensearchproject/opensearch:2.12.0
  • mongo:6.0.14
  • graylog/graylog:6.0.4-1