Nodes with too long GC pauses

Graylog Central (peer support)
1

Hi guys,

we are running a rather small Graylog server (between 1,5 and 3 million messages a day) and for the second time in about 30 days I got greeted by this message:

Nodes with too long GC pauses 

There are Graylog nodes on which the garbage collector runs too long. Garbage collection runs should be as short as possible. Please check whether those nodes are healthy. (Node: *gibberish*, GC duration: *1122 ms*, GC threshold: *1000 ms*)

I checked the java.options for ES and it’s set to 16 GB RAM (both Xms and Xmx) which is half of the physical RAM available. Searching aroud I found this topic: https://community.graylog.org/t/garbage-collection-runs/22868/3 but output_batch_size is already set to 500.

Installed version:
ES: 7.10.2
Graylog: 4.3.8 (will be updated in 2 weeks to latest version)
Java: openjdk 17.0.5

So how can I check that the node is healthy, please? According to the API the status is green.
And is there maybe anything else I can do to fix this warning? Except for getting a faster machine. :smiley:

Any help is much appreciated.
Filisimus

2

I have not seen this message at all so far. This is interesting.

I only have a very unlikely idea: did you get the right settings to increase your java heap? Does the node show 16GB in total in the node overview?

3

Interesting… in the jvm.options config file for ES it is configured to use 16 GB:

https://imgur.com/a/gdqZ7jT (using imgur here because new users can only have 1 embedded image)

But if I check the node itself, it only says 1 GB of heap memory?

node_heap_memory
node_heap_memory1869×741 65.5 KB

Can anyone point out my mistake, please?

But if I remember correctly I read that more heap memory equals longer GC time so increasing the heap memory would not help with the initial problem.

Cheers
Filisimus

4

Hey Filisimus, the nodes that error message is referring to appears to be your Graylog nodes. Use the default file locations of the Graylog files here to edit the “JVM Settings”. You will see Xms and Xmx arguments within the long string within that file that you can set to your desired amount of heap for your Graylog nodes. After doing this, you need to fully reboot the hosts Graylog is installed on to restart the JVM with the new arguments. This should resolve your issue.

2 Likes
5

Thanks a lot Joe. :slight_smile: I will toss in another 3 GB and pray that the Graylog node will be happy then.

Cheers
Filisimus

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.