Graylog crash after CPU and HDD Spikes

Graylog Central (peer support)
1

Hi All,

Newish to Graylog and something funny has started to happen.

Hoping the screenshot comes out ok. I’ve done one so all of the graphs are init.

And two others showing the same info but hopefully comes out a little better.

So something happens, where the CPU spikes to 100% the DISK I/O goes to 2.4G and then there is a memory pressure stall spike and the Memory usage drops from 15.5GB to about 8GB.

The logs have gone crazy, but i can’t see what i should be looking for.
Any suggestions on where to look please?

Thanks - Fletch

graylog
graylog1724×580 125 KB

graylog 2
graylog 21613×552 107 KB

graylog 3
graylog 31637×554 103 KB

2

Hello,

4 CPU seems low.

How many GB have you set for the Graylog and OpenSearch JVM ?

What is the amount of logs ingested ? (logs/second)

Is OpenSearch/DataNode installed on the same host as Graylog ?

Did you check the server.log ?

3

Thanks for the reply.
4 CPU’s is ok in normal use, but i don’t see is what’s causing the CPU to spike.
I’ve recently seen this happen at 3am in the morning, it was fine at 2am and nothing had changed.

It’s only got 2 firewalls and 8 servers (syslog) going into it.

Logs per second are max 45 that i’ve seen.

The VM is 16GB, 8GB for Datanode. All on the same server as per the docker examples from the graylog site.

Hope this helps