Problems recovering after disk space

Graylog Central (peer support)
1

After recovering using the existing topics I run into a different problem. Everything looks like working but no messages are processed.

This is what I could find so far:

NODES
There is 1 active node
2a3767e6 / graylog In 122 / Out 0 msg/s.
The journal contains -2,736,081,861 unprocessed messages in 1 segment. 45 messages appended, 0 messages read in the last second.
Current lifecycle state: Running
Message processing: Enabled
Load balancer indication: ALIVE

I think the problem is here because of the - (minus) and the number is decreasing but very slowly.

Elasticsearch LOG file:
[2018-09-11T10:55:57,851][INFO ][o.e.c.r.a.DiskThresholdMonitor] [hfirGFa] low disk watermark [85%] exceeded on [hfirGFaxQV22WFbivTGK2g][hfirGFa][/var/opt/graylog/data/elasticsearch/nodes/0] free: 29.6gb[10%], replicas will not be assigned to this node

DF result
/dev/sdb1 296G 251G 30G 90% /var/opt/graylog/data/elasticsearch/nodes

2

it looks like your journal is corrupt.

Stop Graylog, delete the Journal (all messages will be lost that are inside the journal), start Graylog again.

Give your Elasticsearch more storage - or raise the low and high watermark values.

3

Thank you,

How do I increase the watermark values?

4

I would always try to search with your favorit search engine …

You might have some hits in the documentation of elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.