Hello,
I have a Graylog server fetching logs from AKS nodes and it keeps rebooting every minute.
Please let me know if I have do any changes related to config.
Below is the error:
@docgyan
Hello,
Your running out of Disk space on your Graylog server. Also it looks like your elasticsearch either can not keep up due to the amount of messages and/or lack of resources like CPU.
To sum it up I would highly suggest increase the volume on the Graylog server.
You probably have a misconfiguration in your server.conf file.
The journal is over flowing and you probably lossing logs at this point.
Thats what I see from what you posted.
Steps:
If this is a Virtual machine shut it down and more disk space.
NEXT
Add a some more vCPU’s and start your Virtual machine back up.
NEXT
Go into your Graylog configuration file and adjust the following.
lets say you have 6 vCPU cores you could make these ajustments.
processbuffer_processors = 3
outputbuffer_processors = 2
inputbuffer_processors = 1
It probably going to take a while to ingest all those messages.
Hope that helps
2 Likes
@gsmith I added 500GB space, 4Vcpus, 16GB RAM and performed the config changes as mentioned related to buffer. Currently am facing below issue:
My server.conf looks like below:
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = fc9b63a79bfd52787b6e93de9befd76acd8971384e3e886763d21800bxxxxxxx
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_publish_uri = http://10.x.x.x:9000/
http_external_uri = http://10.x.x.x:9000/
http_enable_cors = true
elasticsearch_hosts = http://10.x.x.x:9200
elasticsearch_connect_timeout = 10s
elasticsearch_socket_timeout = 60s
elasticsearch_max_total_connections = 200
elasticsearch_max_total_connections_per_route = 20
elasticsearch_max_retries = 2
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_size_per_index = 1073741824
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 3
outputbuffer_processors = 2
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 6553
inputbuffer_processors = 1
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
Hello,
BufferSize must be the power of 2
That would be these as shown below.
ring_size = 65536
inputbuffer_ring_size = 65536
Dont forget to restart graylog service.
Check your graylog log file after restarting it, or maybe tail’ing it for error’s or warnings.
Hope that helps
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.