Journal Contains Unprocessed Messages

Graylog Central (peer support)
1

My instance of Graylog (2.3) isn’t processing messages. This issue has come out of no where, we haven’t changed any config etc. so i’m unsure as to why this issue has arisen.

I have tried restarting: Graylog, Elasticsearch and Mongodb.

I have also deleted the message journal and then restarted everything.

I have checked the logs and there isn’t anything in there at all.

I used netstat to ensure that ES was working and check the GL config to ensure that it was pointed in the right direction of ES.

Not really sure what I can do now, any ideas?

G

2

The issue seems to have resolved itself.

This seems completely unrelated but I deleted the server.log file and the created a new (empty) one, then restarted Graylog twice and messages are now being processed.

G

1 Like
3

I found this by accident also. Purging the log file and it worked fine after. Seemed kinda random but has been working for me.

1 Like
4

This worked for me as well. Had 600k messages in journal on master node only.
(I have 3 graylog nodes in a cluster behind haproxy the other 2 nodes had 10-40 messages in the journal.)
Stopped graylog, renamed /var/log/graylog-server/server.log and created a new empty file.
Started graylog.

Journal on start showed 300k messages which decreased to 30 messages in about 10 minutes and stayed there.
So this definitely fixes the issue but, I don’t understand why…

1 Like
5

I have found deleting the journal seems to fix this issue the majority of the time, the below Graylog forum post goes into the specifics of this.

https://community.graylog.org/t/after-disk-space-issue-no-out-messages-help/1766/6

Regards,

G

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.