Graylog journal utilization and index error

[seighalan (alireza)]

Hi everyone

i installed graylog 5.2 on a G9 Server for a datacenter. gernerally performance is ok but after some days i get some errors about journal utilization and get index error.

OS: Rocky 9.2
Graylog: 5.2
Server: HP G9 (128GB RAM and 56 cores)

cat /etc/graylog/server/server.conf

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = 0DxGQa…
root_password_sha2 = 7c3a6…
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.1.10.11:9000
stream_aware_field_types=false
disabled_retention_strategies = none
allow_leading_wildcard_searches = false
allow_highlighting = false
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 40
outputbuffer_processors = 20
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 4
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000

cat /etc/sysconfig/graylog-server

GRAYLOG_SERVER_JAVA_OPTS=“-Xms40g -Xmx40g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow”
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true”
GRAYLOG_SERVER_JAVA_OPTS=“$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true”
GRAYLOG_SERVER_ARGS=“”
GRAYLOG_COMMAND_WRAPPER=“”

node status image

index error image

notification error

[Joel_Duffield]

You have a problem on message storage into opensearch. From the sounds of it your index has grown huge, can you post your all your indices, and the rotation settings on them. It’s either not rotating when it should, or not rotating often enough.

[seighalan (alireza)]

Hi Joel

my disk usage in /var/lib/elasticsearch is high (2TB usage from 4.5TB)

curl -X GET “localhost:9200/_cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb”

[Joel_Duffield]

You should change the default index set from a 20 day rotation to time size optimized and then set your retention to the length you want. I’m sure those indices are huge, and that’s bad for many different reasons.

[seighalan (alireza)]

thanks for your kind of help