ElasticSearch high watermark, how to move to another drive

1. Describe your incident:
Got the notice stating elasticsearch high watermark reached… I know it says not to necessarily worry about it, but while looking into things, I noticed either elastic or the database (mongo, right?) is apparently not running where I want it to, and I would like to fix that now, early on before I add a bunch of agents…
Clearly, I’ve very new to graylog - A previous sysadmin started setting it up a year ago and quit (the company) before actually getting it running as it turns out. I basically started from the ground up again, edited the 2 agents he had tried all up and running and talking to the “new” graylog server. Pretty much followed the guide for a SUSE install (another downside, that was ‘his’ choice, I’m no master at all, but never used SUSE, only CentOS primarily)…
Problem is, server drives are configured as such:
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 4096 4 4092 1% /dev
tmpfs 65700904 4 65700900 1% /dev/shm
tmpfs 26280364 18596 26261768 1% /run
tmpfs 4096 0 4096 0% /sys/fs/cgroup
/dev/sda2 41942016 41049468 709604 99% /
/dev/sda2 41942016 41049468 709604 99% /.snapshots
/dev/sda2 41942016 41049468 709604 99% /boot/grub2/i386-pc
/dev/sda2 41942016 41049468 709604 99% /boot/grub2/x86_64-efi
/dev/sda2 41942016 41049468 709604 99% /opt
/dev/sda2 41942016 41049468 709604 99% /root
/dev/sda2 41942016 41049468 709604 99% /srv
/dev/sda2 41942016 41049468 709604 99% /var
/dev/sda2 41942016 41049468 709604 99% /tmp
/dev/sda2 41942016 41049468 709604 99% /usr/local
/dev/sda1 522736 5212 517524 1% /boot/efi
/dev/sda3 2637499628 3019752 2634479876 1% /home
tmpfs 13140180 60 13140120 1% /run/user/1000
tmpfs 13140180 28 13140152 1% /run/user/0
/dev/sdb1 109378838764 108982836 109269855928 1% /db

where /db3 was supposedly designated to be the database drive
Knowing little to nothing about gray/elastic/mongo work, I’m not sure how to proceed…
Options I can think of include:

  1. If Mongodb is the heart of gray and elastic, move mongo to the 2.5T drive instead of sda2 which is supposed to be just the OS
    1a. How would I go about that? Could I create a mongo “user” folder in home and somehow move the mongo db to that folder?
    1b. Should I split the 2.5T drive to /home and (create something?) like /database that again would somehow have the mongodb moved to it?
  2. If 1 is not accurate, what would be recommended at this point to “make it right” and get graylog (and it’s components) using the 2.5T drive?

2. Describe your environment:

  • OS Information:
    SLES 15
  • Package Version:
    5.0.7+7758557, codename Noir
  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
Just verified the 95% root drive usage and observed the 2.5t sitting pretty much empty

4. How can the community help?
As mentioned, I’m new at this, took over for another who knew things but apparently struggled to get thiss running. I’m thrilled I got it installed, configured and accepting inputs, but not sure how to fix the fact that by default it’s not using the storage drive (which I get, at some point I have to tell it to use it, but not sure how, and wasn’t prompted during the initial install/setup process). Help and suggestions are highly welcome.

For reference, this graylog server looks to be getting about 10-15+ “inputs” (data sources), which I plan on grouping as “Stream” to keep things orderly (IE Domain Controllers will be grouped in a Stream, SCCM servers in a stream etc…) That sound like a good way to “group” things? (nowhere does it mention grouping, but that’s how I interpreted “streams” to be).

Thanks so much in advance for any and all assistance,

Edit: as an unimportant update, I just found this machine actually does have a massive drive array that apparently was never set up, so won’t be touching the 2.5tb /home drive, lol… I created the massive drive as /dev/sdb1 aka /db, but need to know how/what to move/configure to utilize the /db folder!

One of the fun things about graylog is you are now also a mongodb and elasticsearch/opensearch administrator :slight_smile:

If you are interested in moving your data around, some helpful resources:


MongoDB doesn’t (or really shouldn’t) consume much disk space. It holds configuration data but does not store any log data which accounts for the majority of disk usage of a graylog cluster. (the exception to this would be the auditlog, but i’m assuming you are using Graylog Open Source which doesn’t utilize this feature).

You can configure the database path using the storage.dbPath property in the mongod.conf config file, which i believe defaults to /etc/mongod.conf

Be sure to make sure your folder owner and permissions are kept the same.

Indexers (either Elasticsearch or OpenSearch

The indexer is the primary data store for your log data and as such will make up almost the entirety of your disk usage for a graylog cluster.

Both Elasticsearch or OpenSearch also have configurable data paths.


Using path.data.

See Configuring Elasticsearch | Elasticsearch Guide [7.10] | Elastic for config file locations.

Be sure to make sure your folder owner and permissions are kept the same.


Similar to elasticsearch, you can also configure the data path using path.data attribute.

OpenSearch config file defaults to /etc/opensearch/opensearch.yml

Be sure to make sure your folder owner and permissions are kept the same.

General Tips about disk usage

While logs can consume a lot of disk usage, you can use retention strategies to make disk usage more predictable and even configure it so you don’t have logs take up more than X% of disk space (say 90%).

You’ll want to use a ‘Index Size’ rotation strategy and set the size and number of indices below what your disk space allows.

Here is a blog post that touches on this, although doesn’t specifically talk about the ‘Index Size’ rotation strategy. Log Indexing and Rotation for Optimized Archival in Graylog

Hope that helps!

1 Like

Wanted to say thanks! moving the ElasticSearch to the massive array not only went successfully, but everything is working great now!

Time for more reading about inputs and getting a bunch of servers online and filling the space (haha) I’ll be back with more questions I’m sure, but really appreciate your accurate information!!

  • Rooster

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.