1. Describe your incident:
Got the notice stating elasticsearch high watermark reached… I know it says not to necessarily worry about it, but while looking into things, I noticed either elastic or the database (mongo, right?) is apparently not running where I want it to, and I would like to fix that now, early on before I add a bunch of agents…
Clearly, I’ve very new to graylog - A previous sysadmin started setting it up a year ago and quit (the company) before actually getting it running as it turns out. I basically started from the ground up again, edited the 2 agents he had tried all up and running and talking to the “new” graylog server. Pretty much followed the guide for a SUSE install (another downside, that was ‘his’ choice, I’m no master at all, but never used SUSE, only CentOS primarily)…
Problem is, server drives are configured as such:
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 4096 4 4092 1% /dev
tmpfs 65700904 4 65700900 1% /dev/shm
tmpfs 26280364 18596 26261768 1% /run
tmpfs 4096 0 4096 0% /sys/fs/cgroup
/dev/sda2 41942016 41049468 709604 99% /
/dev/sda2 41942016 41049468 709604 99% /.snapshots
/dev/sda2 41942016 41049468 709604 99% /boot/grub2/i386-pc
/dev/sda2 41942016 41049468 709604 99% /boot/grub2/x86_64-efi
/dev/sda2 41942016 41049468 709604 99% /opt
/dev/sda2 41942016 41049468 709604 99% /root
/dev/sda2 41942016 41049468 709604 99% /srv
/dev/sda2 41942016 41049468 709604 99% /var
/dev/sda2 41942016 41049468 709604 99% /tmp
/dev/sda2 41942016 41049468 709604 99% /usr/local
/dev/sda1 522736 5212 517524 1% /boot/efi
/dev/sda3 2637499628 3019752 2634479876 1% /home
tmpfs 13140180 60 13140120 1% /run/user/1000
tmpfs 13140180 28 13140152 1% /run/user/0
/dev/sdb1 109378838764 108982836 109269855928 1% /db
where /db3 was supposedly designated to be the database drive
Knowing little to nothing about gray/elastic/mongo work, I’m not sure how to proceed…
Options I can think of include:
- If Mongodb is the heart of gray and elastic, move mongo to the 2.5T drive instead of sda2 which is supposed to be just the OS
1a. How would I go about that? Could I create a mongo “user” folder in home and somehow move the mongo db to that folder?
1b. Should I split the 2.5T drive to /home and (create something?) like /database that again would somehow have the mongodb moved to it? - If 1 is not accurate, what would be recommended at this point to “make it right” and get graylog (and it’s components) using the 2.5T drive?
2. Describe your environment:
- OS Information:
SLES 15 - Package Version:
5.0.7+7758557, codename Noir - Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
Just verified the 95% root drive usage and observed the 2.5t sitting pretty much empty
4. How can the community help?
As mentioned, I’m new at this, took over for another who knew things but apparently struggled to get thiss running. I’m thrilled I got it installed, configured and accepting inputs, but not sure how to fix the fact that by default it’s not using the storage drive (which I get, at some point I have to tell it to use it, but not sure how, and wasn’t prompted during the initial install/setup process). Help and suggestions are highly welcome.
For reference, this graylog server looks to be getting about 10-15+ “inputs” (data sources), which I plan on grouping as “Stream” to keep things orderly (IE Domain Controllers will be grouped in a Stream, SCCM servers in a stream etc…) That sound like a good way to “group” things? (nowhere does it mention grouping, but that’s how I interpreted “streams” to be).
Thanks so much in advance for any and all assistance,
rooster
Edit: as an unimportant update, I just found this machine actually does have a massive drive array that apparently was never set up, so won’t be touching the 2.5tb /home drive, lol… I created the massive drive as /dev/sdb1 aka /db, but need to know how/what to move/configure to utilize the /db folder!
