Graylog DataNode Storage – Can Wasabi S3-Compatible Storage Be Used for Hot Data and Archiving?

Graylog Central (peer support)
, ,
1

Hi Graylog Community,

We are currently evaluating our Graylog storage architecture and would appreciate some guidance from the community.

Our current setup has Graylog DataNodes using AWS EFS for data storage. The environment has grown to approximately 40 TB, so we are looking at alternative approaches for both hot data access and long-term archiving.

We understand that Graylog supports AWS S3-related storage capabilities, and we are considering Wasabi as an alternative object-storage backend because it provides an S3-compatible API.

Our questions are:

  1. Can Wasabi S3-compatible storage be used with Graylog/DataNodes for storing or archiving Graylog data?
  2. If technically possible, what would be the recommended architecture? For example, keeping recent/hot data on local EBS/EFS and moving older data to Wasabi for archival.
  3. Are there any performance, compatibility, or operational limitations when using Wasabi instead of native AWS S3?
  4. Are there any specific Graylog features that depend on AWS S3 specifically and therefore would not work with an S3-compatible provider such as Wasabi?
  5. What are the main pros and cons of using Wasabi for this use case, particularly for an environment with approximately 40 TB of data?
  6. Most importantly, does using S3-compatible object storage for Graylog require a Graylog Enterprise subscription, or can this architecture be implemented with the Graylog Open/Community version?

We are primarily interested in reducing the cost and complexity of our current EFS-based storage while maintaining reasonable access to recent/hot data and having a cost-effective long-term archive.

Any recommendations or real-world experience with Graylog + Wasabi would be greatly appreciated.

Thanks!

2

Hey @Golobolus,

Data tiering within Graylog is currently only available within the Enterprise product and the compatible filesystems can be found on the docs page here. Wasabi is not explicitly mentioned and while it may currently work, were a future update to Graylog to contain an update to the S3 library this could break compatibility which would be a nightmare.

So while ill advised but technically possible it would still require an enterprise license. I believe you could utilise Opensearch ISM to implement a data tiering system outside of Graylog but be careful the indices are not renamed in the process as Graylog relies on knowing the alias/name etc to read data.