Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
Hi I’m really a newbie to Linux and Graylog, this has mostly been set up and maintained by interns, but now I’ve got a problem I haven’t been able to fix on my own. This was previously working just fine.
1. Describe your incident:
Recently I’ve been getting error "Journal utilization is too high
Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit"
I also can’t see/find anything in graylog anymore (all streams at throughput = 0 msg)
2. Describe your environment:
-
OS Information:
Debian 11 on Hyper-V VM
351 GO disk
10 GO memory -
Package Version:
Graylog 5.1.8+507d172 -
Service logs, configurations, and environment variables:
I am rotating to keep data for 1 year, I’m monitoring about 6 servers and our firewall is all.
Everything (mongo, elastic, graylog) is all on the same VM.
elastic stack log says this: [2024-03-11T23:32:48,523][WARN ][o.e.c.r.a.DiskThresholdMonitor] [SRV-011-VM] high disk watermark [90%] exceeded on [jgKl9_TxT5SGFGvLe84uhw][SRV-011-VM][/var/lib/elasticsearch/nodes/0] free: 28.2gb[8.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
last changes:
-As this was set up by an intern, recently had to do some work on the host machine as I had disk size issues on C: ( so things got temporarily moved then put back and turned back on (VM are all on D:)
-I added “Windows sucessful logon local” with rules/pipelines to check for logons outside office hours and send alert, this was my first time doing this. I do think it was working when I did this (about 2 months ago)
I really don’t monitor graylog this closely, I just make sure the web console is reachable. We’ve added this to meet client requirements, even though I find it useful sometimes, I’m a one person IT departement so I’m not logging on everyday, so not sure when it stopped logging, saw the journal error two weeks ago and had given it more diskspace to see if that would help but doesn’t look like it did.
3. What steps have you already taken to try and solve the problem?
I tried restarting mongodb, elasticsearch and graylog
Checked “Journal size” is commented out on my server.conf file, so I assume that it is currently at the default size?
Found a command “df -h” that shows me this:
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
udev 4,9G 0 4,9G 0% /dev
tmpfs 995M 1,2M 994M 1% /run
/dev/sda2 339G 296G 29G 92% /
tmpfs 4,9G 0 4,9G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
/dev/loop3 64M 64M 0 100% /snap/core20/2105
/dev/loop1 106M 106M 0 100% /snap/core/16574
/dev/loop0 32M 32M 0 100% /snap/glpi-agent/x1
/dev/loop2 64M 64M 0 100% /snap/core20/2182
/dev/loop4 106M 106M 0 100% /snap/core/16202
/dev/sda1 511M 5,8M 506M 2% /boot/efi
tmpfs 995M 72K 995M 1% /run/user/115
tmpfs 995M 60K 995M 1% /run/user/1000
4. How can the community help?
Any help on how I get Graylog fonctional again would be appreciated! I’ll admit I don’t really know what I’m doing and haven’t been able to find training. Thank you!
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]