We’re currently doing some load testing to find out how Graylog reacts to an overload situation.
Setup: graylog sidecar with filebeat sends logs to graylog with kafka journal enabled and elastic search.
Once we send messages faster than Elasticsearch can injest the kafka journal fills up and and starts to discard messages.
Questions:
can we have a Graylog setup where Graylog slows down injestion from clients and doesn’t discard messages? Can this be achieved by disabling the Kafka journal?
Wishes:
I am missing a chapter describing how Graylog reacts to high load. I found out about discarding messages only by reading the configuration file.
I’d like to know if Graylog implements for example back-pressure strategies (when using file-beat this sounds like a reasonable alternative on the client side to discarding messages on the Graylog server side)
You need to size the journal according to your needs, that could include a high amount of memory and a long time to hold the data. The primary goal should be for elasticsearch to be able to handle your normal load without any problems. That the journal is only for maintenance and high load situations.
Graylog has not back-pressure strategies as most protocols did not have a back channel, if you want to be sure to loose no messages, implement buffers in between everything. That would include a message cue like rabbitMQ or kafka between your sending server and Graylog.
So what will happen when I disable the kafka journal? Will the messages fill up the RAM, or will Graylog
start to discard messages when a certain number of messages is stored in RAM?
When I setup a queue with rabbitMQ or kafka, will Graylog only retrieve messages at the rate it can forward to Elasticsearch?
