# Graylog not processing messages after crash (ran out of space) **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [April 2, 2020, 2:10pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786 "2020-04-02T14:10:18Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![tnacnud1](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tnacnud1/32/1116_2.png) [@tnacnud1](https://community.graylog.org/u/tnacnud1)\ **Post date:** [April 2, 2020, 2:10pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/1 "2020-04-02T14:10:18Z")
Good morning. I am hoping someone can help me shed light on what I need to look at there. This morning the root partition on my Graylog server ran out of space due to a java heap dump being placed on the root directory. I deleted the dumps and restarted elasticsearch, mongod, and graylog-server. Some time passed and I noticed the streams were no longer getting messages. I then restarted the physical server and let everything come back online. All services are running (and INPUTs even show messages being received), but there are no messages in the streams. So I started looking through the logs and I cannot find anything to indicate any problems. The Sources are showing as empty and I’m just not sure what to look at. I did rotate the active index, and still nothing is happening. I did notice this in my “node” section: Current lifecycle state: Failed Message processing: Disabled Load balancer indication: DEAD ---
**Author:** ![tnacnud1](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tnacnud1/32/1116_2.png) [@tnacnud1](https://community.graylog.org/u/tnacnud1)\ **Post date:** [April 2, 2020, 2:31pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/2 "2020-04-02T14:31:13Z")
I did go ahead and “start processing” on the node (since it’s the only node I have). It looks like it is processing the messages in the journal, but it’s filling up quickly. Does anyone know what happens when the journal becomes full? ---
**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\ **Post date:** [April 2, 2020, 2:36pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/3 "2020-04-02T14:36:58Z")
I’m not sure if Elasticsearch does this everytime, but it sounds like Elasticsearch made all of its indices read only. Check this thread on some insight. > [@Graylog index rotation not working](https://community.graylog.org/t/graylog-index-rotation-not-working/11404): > > Hello, Previously my index was set to store logs for about 4 years. That was my first, initial, and not quite a wise setting. But it was, and it worked. Unfortunatelly partition where folder /var is mounted (where I think logs from Graylog are stored) is occupied in 55%. I think this causes that index in Graylog won’t accept new logs - it shows " There were 204,800 failed indexing attempts in the last 24 hours." Few days ago I changed index roration period from 4 years to P1W (7d, 7 days) but… In my experience, the journal will fill to about 104%, but then it will start purging messages… oldest first to free up room for the newest messages. you can expand your journal size, but it sounds like once you reenable write on your elasticsearch indices, you should be fine. hth ---
**Author:** ![tnacnud1](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tnacnud1/32/1116_2.png) [@tnacnud1](https://community.graylog.org/u/tnacnud1)\ **Post date:** [April 2, 2020, 2:43pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/4 "2020-04-02T14:43:23Z")
Thanks for the response. I did finally get it to work. It’s bizarre- I had to stop graylog-server then I had to manually delete the journal-\> for me it was ’ /var/lib/graylog-server/journal/’. Then once I started graylog-server again the message processing is working again… I hope this can help someone else out in the future. The root of my problem though is that Java is automatically dumping its HEAP onto my root dir which is very small. I have my heap configured at 80 GB. I need to change the directory it dumps to… ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [April 6, 2020, 2:27pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/5 "2020-04-06T14:27:29Z")
he @tnacnud1 why did you have configured the java heap to 80GB? What is the idea behind that? In addition it is very likely that you Graylog journal was corrupt - what happens is the configured space is not exclusive available for Graylogs journal ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [April 20, 2020, 2:27pm UTC](https://community.graylog.org/t/graylog-not-processing-messages-after-crash-ran-out-of-space/14786/6 "2020-04-20T14:27:33Z")
Closed on April 20, 2020, 2:27pm UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.