I’ve run 10TB worth elastic indices in retention on a single server + Graylog with slightly higher specs than listed. I wouldn’t recommend doing that, but it’s possible.
You should be fine with the specs listed.
All application running in docker container and sending logs via logstash GELF UDP. i wanted to make graylog available 24x7. so my question is
What if network brake between logstash and Load Balancer. can we set up some buffer system in logstash to save log until the network is available again.
Current log entries is 40 Million logs per day so what if network is not available for 6 hours and where i can store this 10 Millions logs ?
can we use kafka but again the question remain the same for network outage
… but logstash has no buffer, if logstash can’t push messages somewhere this messages are lost. You can place some buffers like kafka or AMQP in the network but if that is not reachable for logstash again, messages are lost.
Theoretically, you could direct the logstash logs to the local syslog, so as to use it as a transfer point. It can do what you want:
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
But I’m not sure, if Syslog can receive GELF messages in reality.