Graylog server rebooting frequently

Hello,

I have a Graylog server fetching logs from AKS nodes and it keeps rebooting every minute.
Please let me know if I have do any changes related to config.

Below is the error:

@docgyan
Hello,

Your running out of Disk space on your Graylog server. Also it looks like your elasticsearch either can not keep up due to the amount of messages and/or lack of resources like CPU.

To sum it up I would highly suggest increase the volume on the Graylog server.
You probably have a misconfiguration in your server.conf file.
The journal is over flowing and you probably lossing logs at this point.
Thats what I see from what you posted.
Steps:
If this is a Virtual machine shut it down and more disk space.
NEXT
Add a some more vCPU’s and start your Virtual machine back up.
NEXT
Go into your Graylog configuration file and adjust the following.
lets say you have 6 vCPU cores you could make these ajustments.

processbuffer_processors = 3
outputbuffer_processors = 2
inputbuffer_processors = 1

It probably going to take a while to ingest all those messages.
Hope that helps

2 Likes

@gsmith I added 500GB space, 4Vcpus, 16GB RAM and performed the config changes as mentioned related to buffer. Currently am facing below issue:

My server.conf looks like below:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = fc9b63a79bfd52787b6e93de9befd76acd8971384e3e886763d21800bxxxxxxx
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_publish_uri = http://10.x.x.x:9000/
http_external_uri = http://10.x.x.x:9000/
http_enable_cors = true
elasticsearch_hosts = http://10.x.x.x:9200
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
elasticsearch_max_total_connections = 200
elasticsearch_max_total_connections_per_route = 20
elasticsearch_max_retries = 2
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_size_per_index = 1073741824
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 3
outputbuffer_processors = 2
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 6553
inputbuffer_processors = 1
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Hello,

BufferSize must be the power of 2

That would be these as shown below.

ring_size = 65536
inputbuffer_ring_size = 65536

Dont forget to restart graylog service.
Check your graylog log file after restarting it, or maybe tail’ing it for error’s or warnings.
Hope that helps