High CPU Usage Graylog 3.3.9

[staybox (Andrey)]

Hello!

We have a graylog and it consumes a lot of CPU time. Always 100%.
Graylog version: 3.3.9
Java version: java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64
VM has 16 cores and 48 GB RAM

What did you try to do?

1. Checked the config file of the greylog. On the processor, everything is set correctly, as far as I understand.
2. Updated Graylog itself to version 4.2 - did not help
3. Updated Java version for graylog version 3.3.9 - did not help (tried to use version 11)
4. For version 4.2, I used three different versions of Java - 1.8, 11, 17 - did not help.
5. Tried inscrese cores from 16 to 40 but nothing.
6. Disabled inputs - but nothing.
   Could you suggest where else to look?

top -H:

cat /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = SECRET
root_username = root
root_password_sha2 = SECRET
root_timezone = Europe/London
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = 0.0.0.0:9000/api/
rest_transport_uri = IP:9000/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
web_enable = true
web_listen_uri = 0.0.0.0:9000/
elasticsearch_hosts = server01:9200,server02:9200
elasticsearch_max_total_connections = 20
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 1000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 12
outputbuffer_processors = 4
processor_wait_strategy = yielding
ring_size = 2097152
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = yielding
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 72h
message_journal_max_size = 32gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = smtp.domain.com
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog]
transport_email_from_email = SERVER@domain.com
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Thanks.

[gsmith]

Hello && Welcome @staybox

From the title of this post and comparing it to your setting and resources.
I see that JAVA is not the option in this case.
First, What i see is these settings since were talking about High CPU Usage.

processbuffer_processors = 12
outputbuffer_processors = 4
inputbuffer_processors = 2

Those combined equal are 18 CPU threads created, It is advisable to increase the amount of CPU cores. For example If you don’t need that many Processor Buffer I would decrease it this would depend on how many messages are being ingested. A ideal setting would be is the combine buffer setting would be less then the amount of CPU cores this device has, You need to insure that a couple CPU cores is left for the OS.

Next, Bad GROK/REGEX extractors or pipelines can be the culprit this will defiantly jack up the CPU usage and/or memory.

And last, Your version of Graylog

@staybox

Graylog version: 3.3.9

Does not match the version of Graylog Configuration file. You need to look at the documentation here.

• server.conf

The Graylog configuration your using is for 2.4 NOT 3.3 or 4.2. you may want to recheck that.

[staybox (Andrey)]

Hello,

Many thanks for your answer.

We changed this parameters in server.conf file (from yielding to blocking and CPU usage don’t high now.):
processor_wait_strategy = blocking
inputbuffer_wait_strategy = blocking

I think that changed parameters solved high CPU usage problem.