Graylog not processing messages after crash (ran out of space)

Graylog Central (peer support)
1

Good morning. I am hoping someone can help me shed light on what I need to look at there. This morning the root partition on my Graylog server ran out of space due to a java heap dump being placed on the root directory. I deleted the dumps and restarted elasticsearch, mongod, and graylog-server. Some time passed and I noticed the streams were no longer getting messages.

I then restarted the physical server and let everything come back online. All services are running (and INPUTs even show messages being received), but there are no messages in the streams. So I started looking through the logs and I cannot find anything to indicate any problems. The Sources are showing as empty and I’m just not sure what to look at.

I did rotate the active index, and still nothing is happening.

I did notice this in my “node” section:

Current lifecycle state:
Failed
Message processing:
Disabled
Load balancer indication:
DEAD

2

I did go ahead and “start processing” on the node (since it’s the only node I have). It looks like it is processing the messages in the journal, but it’s filling up quickly. Does anyone know what happens when the journal becomes full?

3

I’m not sure if Elasticsearch does this everytime, but it sounds like Elasticsearch made all of its indices read only.

Check this thread on some insight.

In my experience, the journal will fill to about 104%, but then it will start purging messages… oldest first to free up room for the newest messages. you can expand your journal size, but it sounds like once you reenable write on your elasticsearch indices, you should be fine.

hth

1 Like
4

Thanks for the response. I did finally get it to work. It’s bizarre- I had to stop graylog-server then I had to manually delete the journal-> for me it was ’ /var/lib/graylog-server/journal/’. Then once I started graylog-server again the message processing is working again…

I hope this can help someone else out in the future. The root of my problem though is that Java is automatically dumping its HEAP onto my root dir which is very small. I have my heap configured at 80 GB. I need to change the directory it dumps to…

5

he @tnacnud1

why did you have configured the java heap to 80GB? What is the idea behind that?

In addition it is very likely that you Graylog journal was corrupt - what happens is the configured space is not exclusive available for Graylogs journal

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.