We recently setup Graylog for a cluster of Apache2 servers using piped log messages via /bin/nc
At some point in time, graylog’s listener stopped working and we started filling the apache2 error logs with
piped log program '/bin/nc -u OURSERVER.com 12201' failed unexpectedly errors
Around the same time, our web clusters started dying unexpected deaths and falling off their load ballancers - I suspect because the hung connections to graylog were causing them to fault somehow.
That said, I suspect the failure of a graylog listener in our case was the cause of a cascade failure across our webserver clusters.
With the above in mind, what is best practice to get data from an apache2 webserver cluster to a graylog install in a way that prevents this from happening in the future should the graylog instance “go away”?
Many thanks in advance for the conversation and advice to come!