Journal utilization is too high again

Graylog Central (peer support)
1

Hi all.

I been reading through some previous post about the “subject” in my post but cant seem to find what I need in those.

My graylog installation, testing, keeps failing with out messages. It receives about 30-50 msg per seconds and I dont think that is very high. Its a default installation, out of the box, with one modification listed below.

If I reboot the server it works for a little while then goes back to not processing the msg. Incoming works but not outgoing. It just seems to pile up and I found today I had 500.000.000 msg waiting… Yea I left it for a while.

image

I am not that well versed in linux but not a total beginner either. What would I need to post to get this troubleshooted?

Server info: Virtual in Hyper-V.
CPUx2
4GB RAM
100GB disk

I have noticed that Hyper-V say it demands 5283MB of RAM

I only have one firewall sending logs to it currently as it is a test.

I have extended the JVM memory to 2GB but thats the only tweak I done so far.

All services seem ok as I run the graylog-ctl status command.

Any suggestions would be nice.

Regards
Conny

2

What’s in the logs of your Graylog and Elasticsearch nodes?
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

Is Graylog able to connect to the Elasticsearch cluster?

3

The path to elasticsearch logs cant be found and when I check the graylog logs it gives me a permission denied.

root@graylog:/var/log# ls
alternatives.log    auth.log.2.gz  dist-upgrade  dmesg.3.gz     faillog    kern.log.1     ntpstats     syslog.4.gz  upstart
alternatives.log.1  auth.log.3.gz  dmesg         dmesg.4.gz     fsck       kern.log.2.gz  syslog       syslog.5.gz
apt                 auth.log.4.gz  dmesg.0       dpkg.log       graylog    kern.log.3.gz  syslog.1     syslog.6.gz
auth.log            boot.log       dmesg.1.gz    dpkg.log.1     installer  kern.log.4.gz  syslog.2.gz  syslog.7.gz
auth.log.1          bootstrap.log  dmesg.2.gz    dpkg.log.2.gz  kern.log   lastlog        syslog.3.gz  udev
root@graylog:/var/log# /var/log/elasticsearch/*/current
bash: /var/log/elasticsearch/*/current: No such file or directory
root@graylog:/var/log# /var/log/graylog/*/current
bash: /var/log/graylog/elasticsearch/current: Permission denied
root@graylog:/var/log#
4

You cannot execute log files. You either have to download the files or use a viewer, such as less, to take a look at the logs.

5

Ok… managed to use less and looked through graylog.log file.

[2018-05-31T00:28:01,562][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node
[2018-05-31T00:28:01,562][INFO ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-05-31T00:28:31,564][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node
[2018-05-31T00:29:01,565][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node
[2018-05-31T00:29:01,565][INFO ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-05-31T00:29:31,567][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node
[2018-05-31T00:30:01,570][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node
[2018-05-31T00:30:01,570][INFO ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-05-31T00:30:31,571][WARN ][o.e.c.r.a.DiskThresholdMonitor] [uYUXalB] high disk watermark [90%] exceeded on [uYUXalBhRQWrctZAudACfA][uYUXalB][/var/opt/graylog/data/elasticsearch/nodes/0] free: 4.4gb[4.5%], shards will be relocated away from this node

Those logs seems to dominate it…

I get the disk space seems to be close to running out but what do I do about it? I rather have the logs that comes into the graylog gets treated like circular logging in Exchange so I can set a maximum value and then it just whipes the old stuff. If possible…

6

Then just configure Graylog to do that. See Index sets / Default index set / Edit

Then set the retention settings.

7

Yea, it seems to have done the trick. I lowered the diskusage on the indices and the number of indices and set it to delete once it has to start over. At least the in/out seems to work fine. :slight_smile:

Thank you for the assistance.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.