I hope someone can point me to the right direction on this one. We use 3 Graylog servers and 3 datanodes cluster, and we would like to use it as as a central logging however, a few months ago we had an complete outage of graylog due to corrupted datanode that has replicated across the other datanodes and took down the servers completely.
That means all logs that were coming into our HAProxy to be distributed to graylog have been lost, we lost few hours worth of logs.
That brings up the question - how do we avoid losing logs when graylog is completely down? I was looking as syslog-ng and disk buffering options. I wonder if there are better solutions?
That’s difficult to answer, because it depends on a lot of factors.
If Graylog is running, it logs everything to local disk as a buffer, the Disk Journal. This could be increased for different reasons, temporary storage with backbend problems is one of them.
Logs could be kept longer on devices sometimes. Log shippers can be setup with temporary memory in case of trouble, and store some by default,just as rsyslog does as we see, without offending the timestamps.
One problem is that logs are can be curated specifically for graylog. With ELK-stack a Redis cluster is often used to get capacity in front of the stack for temporary storage and when high volumes are coming in as I have sen at a demo by bol.com on elastic.
how do we avoid losing logs when graylog is completely down
Your best bet is to find ways for the log sources to have their own backlog mechanisms. Beats is a good example of this.
Unfortunately (in that it isn’t terribly helpful for you) the complete answer to your question is to use a Graylog Forwarder, but this does require an enterprise license.
Having the sender hold the logs with Collector Sidecar/beats is preferred. For scenarios where that’s not an option, I’d recommend Kafka. We ended up using Kafka in combination with Vector. Vector provided superior throughput for load balancing logs coming in compared to everything else I tried. And Kafka allowed us to store and then process logs faster during outages and heavy loads.
