We are using graylog cluster of 4 nodes, mongodb RS and separate elasticsearch cluster.
Each graylog server has elasticsearch client-only service, and each graylog server configured to send logs to itself and to other 3 nodes. Sometimes i observe nodes that just stop to send output to elasticsearch: output buffer is 100% full and no messages are going out.
Using tcpdump i have noticed that graylog server does not even tries to send output elasticsearch, gathering messages in its local journal. I don
t see any rejects/denies in tcpdump from the side of elasticsearch, messages just dont go out, while at the same time other cluster members send messages successuly to elasticsearch.
- What logs should i check to try to debug it ? as in log of graylog server nothing appear. as in local journalct and in elasticsearch log
- What configuration parameters should i generally set to fix it ?
graylog version 2.5
elastic version: 5.6.15
relevant configuration (may be i miss something here)
elasticsearch_hosts = http://10.25.100.57:9200,http://10.25.100.87:9200,http://10.25.100.40:9200,http://10.25.100.18:9200
elasticsearch_connect_timeout = 2s
elasticsearch_socket_timeout = 5s
elasticsearch_max_total_connections = 256
elasticsearch_max_total_connections_per_route = 64
elasticsearch_max_retries = 1
output_batch_size = 1024
output_flush_interval = 1
processbuffer_processors = 5
outputbuffer_processors = 7
ring_size = 262144
inputbuffer_ring_size = 16384
inputbuffer_processors = 4
inputbuffer_wait_strategy = blocking
message_journal_enabled = true