STORAGE limit issue since upgrade 7.1 to 7.2 with datanode upgrade to embedded Opensearch 3.7

Hi,

Since the upgrade, the storage of datanode is limited to 50GB.
All the indices are in read-only mode.

Data node got enough space on the zfs system on Ubuntu 22.04.4 LTS

sudo df -h /var/lib/graylog-datanode/opensearch/data/
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/ubuntu_1n1n1w/var/lib 2,5T 1,6T 984G 62% /var/lib

I m using graylog open.

I ve tried to edit mongodb, datanode and graylog conf to expand the storage limit but nothing working.

Is it a bug with zfs, how can i fix it ?

Thx

Hey @veasna,

This only occurred post update?

Is the 50GB of available storage shown in the UI actually reflected when storing logs or a UI bug, are you getting watermark errors within the Data Node log?

Hello,
datanode could have take too much cache size than real storage space to unlock all indices after the upgrade opensearch 2.91.5 to 3.7.

I could fix it with this value in datanode.conf
node_search_cache_size = 800gb
and delete the read only lock in all the indices in opensearch
curl -X PUT "http://localhost:9200/_all/_settings" \
-H ‘Content-Type: application/json’ \
-d ‘{“index.blocks.read_only_allow_delete”: null}’

Graylog is working again after restart graylog-server and grayog-datanode

Thx for your help