Using Amazon EFS storage for the Journal

Graylog Central (peer support)
1

Using Graylog 5.2 on Ubuntu 22.04LTS, with Docker Compose. My inputs are Beat inputs coming from Filebeat that runs on Kubernetes.

I’m mounting an Elastic Filesystem on /data/graylog, and in my docker-compose I’m mounting the host’s /data/graylog to the container’s /usr/share/graylog/data

My journal size is set to a max of 400GB. Usually I use just a few hundred MB, say 200MB. The idea obviously is to allow for the journal to grow a lot so as to cover any crisis that might come up with OpenSearch, but only pay for what I use, not reserve 400GB up front.

When I start the container, I get an error that the available space of about 200MB is not sufficient and another ~400GB are required. For some peculiar reason It sees the existing capacity as the limit. And of course Graylog doesn’t start. In other words, Graylog doesn’t see the file system as capable of extending. This happens only now that I’ve enabled this EFS storage, it was not happening before when the mounted folder on the host was regular EBS storage.

The only workaround is to run rm -rf journal/* on the host and restart the containers. Then Graylog starts fine, but I’ve lost the contents of the journal.

Is there a way out of this? So that I can restart Graylog (for whatever reason it may be necessary) without losing any data?

If this problem comes with EFS, is there perhaps a way to consume and therefore empty the Journal, so that I can then remove it before the restart without losing any data? Can Graylog somehow “wait” for me to restart?

Thanks for any clues.

2

I can understand that perhaps no one is using EFS and therefore no one can help me with my exact problem, but my last question looks simple enough for people who know Graylog well:

*…is there perhaps a way to consume and therefore empty the Journal, so that I can then remove it before the restart without losing any data? *

3

You could use the config flag to skip the pre-flight checks, which may have other consequences, but this would allow Graylog to start even when the journal config is defined as larger than available storage; you might want to test if the journal actually is able to use the storage as it extends through.

skip_preflight_checks = true

1 Like
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.