Single node server - stopped processing messages after disk full incident

Graylog Central (peer support)
1

As subject states: disk ran full at 0% free space. Disk was expanded and graylog-server restarted.
Looking in System > Nodes, I can see that after graylog-server restart the buffers are working like expected for a few seconds, maybe half a minute, appending and reading thousands of messages pr second. THen it changes to 0 messages or perhaps some random small number, same for both buffers, but the unprocessed messages number grows steadily.
I have wiped the journal folder, restarted both graylog-server and the VM multiple times.

What to do?

2

@DavDav
Do you see any errors in the log files (i.e. elasticsearch, graylog)?
Status of the service/s when this happens, are they still running?

3

Not sure what order you did things in, but according to this post by @jan, you should stop graylog, purge the journal, then start graylog. Seems a similar situation to yours.

Also, make sure ES isn’t in a read only state. That can happen with low available space issues.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.