How to increase out messages

this is a chinese user of graylog,i have a question like this :

my cluster has 8 hosts for elastic and 5 hosts for graylog, every host is 128GB RAM and 48 cpus,my graylog config is like this:

is_master = false
node_id_file = /etc/graylog/server/node-id

root_timezone = Asia/Shanghai

allow_leading_wildcard_searches = false
allow_highlighting = true
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 30
outputbuffer_processors = 30
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true

message_journal_dir = /home/richmail/project/graylog-server/journal
message_journal_max_size = 20gb
lb_recognition_period_seconds = 3

mongodb_uri = mongodb://
mongodb_max_connections = 2000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

when the Rotation happend,the out of message is very low,how to resolve this ? please。。。

by the way ,I have 6 indices ,every indice config is
Index prefix:push-mesgdel
Field type refresh interval:30 seconds
Index rotation strategy:Index Time
Rotation period:P1D (1d, a day)
Index retention strategy:Delete
Max number of indices:20

you might need to adjust the configuration for elasticsearch. As your outputbuffer is filled I guess that Elasticsearch is not working proper.

If you share the configuration of Elasticsearch, the JVM Settings and your architecture we might be able to help.

elasticsearch.yml cx_graylog
action.auto_create_index: false es-node-108 /home/richmail/project/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
http.port: 9200 [“”, “”, “”]
discovery.zen.minimum_master_nodes: 2

I just change Xms to 32g about jvm.options

Global inputs

  • bind_address:
  • no_beats_prefix:false
  • number_worker_threads:24
  • override_source:
  • port:8044
  • recv_buffer_size:1048576
  • tcp_keepalive:false
  • tls_cert_file:
  • tls_client_auth:disabled

I use sidecar to control filebeat
filebeat —> graylog(beats)----->es

what kind of storage did you use for elasticsearch?

You might want to create 8 shards that for all indices you have a shard on each elasticsearch server - or even 16 to be able to scale that if needed. Currently you push the load each day to 4 elasticssearch servers only.

thanks, i will try to creat 8 shards

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.