Graylog not processing messages after crash (ran out of space)

[cawfehman]

I’m not sure if Elasticsearch does this everytime, but it sounds like Elasticsearch made all of its indices read only.

Check this thread on some insight.

Graylog index rotation not working

Hello, Previously my index was set to store logs for about 4 years. That was my first, initial, and not quite a wise setting. But it was, and it worked. Unfortunatelly partition where folder /var is mounted (where I think logs from Graylog are stored) is occupied in 55%. I think this causes that index in Graylog won’t accept new logs - it shows " There were 204,800 failed indexing attempts in the last 24 hours." Few days ago I changed index roration period from 4 years to P1W (7d, 7 days) but…

In my experience, the journal will fill to about 104%, but then it will start purging messages… oldest first to free up room for the newest messages. you can expand your journal size, but it sounds like once you reenable write on your elasticsearch indices, you should be fine.

hth