# Journal not Getting processed after ES failure

**URL:** <https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877>\
**Category:** Graylog Central (peer support)\
**Created:** [October 23, 2017, 7:23pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877 "2017-10-23T19:23:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shines2](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shines2](https://community.graylog.org/u/shines2)\
**Post date:** [October 23, 2017, 7:23pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877/1 "2017-10-23T19:23:53Z")

</div>

I had an ES failure during the weekend (ran out of space) which I resolved today.

Once I fixed ES graylog started ingesting the unprocessed logs from my other 30 graylog servers.

However /var was filled up and graylog stopped processing logs.

I then stopped graylog and moved the journal to a dedicated partion and rsynced all the files from /var/lib/graylog/journal to the new location.

I then started both nodes in the graylog cluster but now I have 5 million unprocessed messages.

I found the following post: [After disk space issue - no out messages - help](https://community.graylog.org/t/after-disk-space-issue-no-out-messages-help/1766)

I really don;t want to delete the journal because I’m assuming I will lose all the messages currently being stored within the journal. Is that assumption false?

I also noticed that the journal on node2 is not being populated like it is on node1.

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [October 23, 2017, 7:30pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877/2 "2017-10-23T19:30:02Z")

</div>

> [@shines2](#):
>
> I really don;t want to delete the journal because I’m assuming I will lose all the messages currently being stored within the journal. Is that assumption false?

The assumption is correct. The journal contains all messages which have been received by Graylog but which haven’t been processed and written to the outputs (i. e. Elasticsearch) yet.

Unfortunately I’m afraid you’ll have to delete the journal files as they have likely been corrupted when the system ran out of disk space. ☹

---

<div class="post-metadata">

**Author:** ![shines2](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@shines2](https://community.graylog.org/u/shines2)\
**Post date:** [October 23, 2017, 7:42pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877/3 "2017-10-23T19:42:50Z")

</div>

Thanks for the quick reply (as always 😃slight\_smile:)

Deleting the journal did the trick but I lost 5GB worth of data.

Do you have any recommendations for preventing journal corruption (other than not running out of disk space)?

Luckily I don’t have this system in production yet, but will will be using it for NIST compliance so it’s super important that I don’t lose data.

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [October 23, 2017, 8:34pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877/4 "2017-10-23T20:34:50Z")

</div>

> [@shines2](#):
>
> Do you have any recommendations for preventing journal corruption (other than not running out of disk space)?

That’s currently the only good advice I have as it seems to be the number one reason for a corrupted journal.

If you want, you can chime in on the discussion to the following pull request:

> <https://github.com/Graylog2/graylog2-server/pull/4238>
>
> Users regularly run into problems with their Graylog setups due to the disk jour…nal filling up the whole disk and thus corrupting the journal.
> 
> This change set adds a periodical which checks the available disk space in the journal directory and if it's insufficient, creates an urgent system notification and switches the load-balancer status of the Graylog node to DEAD.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [November 6, 2017, 8:35pm UTC](https://community.graylog.org/t/journal-not-getting-processed-after-es-failure/2877/5 "2017-11-06T20:35:10Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.