Prevent data lost after crash

Graylog
#1

Hello, My graylog crash this weekend (I can’t find cause on log, nothing interesting in logs)
Problem was no log was stored in graylog since last friday 17’pm.

My problem is journal didn’t work.
After restart graylog/ES/mongo, no processing new messages, and no processing old messages from journal.

I tryed to delete .lock after stopping it, but same.

Finaly, I had to remove my /var/lib/graylog2/journal to process new messages again, and list my weekend data.

How may I prevent this lost ?
I’m on Graylog 3.3.3

Thanks

#2

Hello @Le-DOC,

When you say Graylog crashed, what do you mean? The graylog-server service stopped? If so there should be some information in the service log, assuming that the service was able to write the error information to disk. It sounds like your journal was corrupted since deleting it resolved the issue. Does the Graylog host have any errors in the system log relating to disk?

#3

Hello @ttsandrew
Graylog “crashed” by not processing messages. I had nothing since friday.
When I restarted normally with journal process, Input/Output sec should growth a lot to reprocess messages in journal, and gradually count messages raises from crash date, to now.

But this time, nothing. No growth, no gradually message incoming., and no new logs…
I had to delete journal to get back new messages, and lost weekend messages :frowning:

The crash seems be cause of an extractor consuming too much perf (I’m working on it), so no error logs.

#5

did you checked if the journal size that is configured for graylog is exclusive available for graylog @Le-DOC

from my experience what you describe only happens if you are having a journal size configured that is not given and the journal corrupt itself by that.

#6

Here is my configuration:

Configuration

Path:
file:///var/lib/graylog-server/journal/
Earliest entry:
5 minutes ago
Maximum size:
5.0GiB
Maximum age:
12 hours 0 minutes
Flush policy:
Every 1,000,000 messages or 1 minute 0 seconds

I know problem with disk full, but in my case, it wasn’t. I have 8Go free for my /var.

I think I should create a dedicated partition for this journal for security