# Prevent data lost after crash

**URL:** <https://community.graylog.org/t/prevent-data-lost-after-crash/18875>\
**Category:** Graylog Central (peer support)\
**Created:** [February 22, 2021, 8:49am UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875 "2021-02-22T08:49:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Le-DOC](https://avatars.discourse-cdn.com/v4/letter/l/4af34b/32.png) [@Le-DOC](https://community.graylog.org/u/Le-DOC)\
**Post date:** [February 22, 2021, 8:49am UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/1 "2021-02-22T08:49:23Z")

</div>

Hello, My graylog crash this weekend (I can’t find cause on log, nothing interesting in logs)  
Problem was no log was stored in graylog since last friday 17’pm.

My problem is journal didn’t work.  
After restart graylog/ES/mongo, no processing new messages, and no processing old messages from journal.

I tryed to delete .lock after stopping it, but same.

Finaly, I had to remove my /var/lib/graylog2/journal to process new messages again, and list my weekend data.

How may I prevent this lost ?  
I’m on Graylog 3.3.3

Thanks

---

<div class="post-metadata">

**Author:** ![ttsandrew](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@ttsandrew](https://community.graylog.org/u/ttsandrew)\
**Post date:** [February 23, 2021, 9:39pm UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/2 "2021-02-23T21:39:32Z")

</div>

Hello @Le-DOC,

When you say Graylog crashed, what do you mean? The graylog-server service stopped? If so there should be some information in the service log, assuming that the service was able to write the error information to disk. It sounds like your journal was corrupted since deleting it resolved the issue. Does the Graylog host have any errors in the system log relating to disk?

---

<div class="post-metadata">

**Author:** ![Le-DOC](https://avatars.discourse-cdn.com/v4/letter/l/4af34b/32.png) [@Le-DOC](https://community.graylog.org/u/Le-DOC)\
**Post date:** [February 25, 2021, 8:59am UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/3 "2021-02-25T08:59:13Z")

</div>

Hello @ttsandrew  
Graylog “crashed” by not processing messages. I had nothing since friday.  
When I restarted normally with journal process, Input/Output sec should growth a lot to reprocess messages in journal, and gradually count messages raises from crash date, to now.

But this time, nothing. No growth, no gradually message incoming., and no new logs…  
I had to delete journal to get back new messages, and lost weekend messages ☹

The crash seems be cause of an extractor consuming too much perf (I’m working on it), so no error logs.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [February 26, 2021, 6:48am UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/5 "2021-02-26T06:48:43Z")

</div>

did you checked if the journal size that is configured for graylog is exclusive available for graylog @Le-DOC

from my experience what you describe only happens if you are having a journal size configured that is not given and the journal corrupt itself by that.

---

<div class="post-metadata">

**Author:** ![Le-DOC](https://avatars.discourse-cdn.com/v4/letter/l/4af34b/32.png) [@Le-DOC](https://community.graylog.org/u/Le-DOC)\
**Post date:** [February 26, 2021, 12:43pm UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/6 "2021-02-26T12:43:48Z")

</div>

Here is my configuration:

### Configuration

Path:  
file:///var/lib/graylog-server/journal/  
Earliest entry:  
5 minutes ago  
Maximum size:  
5.0GiB  
Maximum age:  
12 hours 0 minutes  
Flush policy:  
Every 1,000,000 messages or 1 minute 0 seconds

I know problem with disk full, but in my case, it wasn’t. I have 8Go free for my /var.

I think I should create a dedicated partition for this journal for security

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [March 12, 2021, 12:44pm UTC](https://community.graylog.org/t/prevent-data-lost-after-crash/18875/7 "2021-03-12T12:44:38Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
