currently I am using Logstash and the GELF plugin to send logfiles to Graylog.
So: Logstash -> GELF -> Graylog
Basically it works, but sometimes Logstash goes crazy and consumes almost all CPU resources - which is particularly bad on the production system. One reason for sure are very complex grok filters.
To take some load off the production machines and make them immune to Logstash hickups I’m thinking about makind a dedicated Logstash cluster.
So something like this:
Filebeat -> Logstash Cluster -> GELF -> Graylog
Question is: Shoud I put a message broker, e.g. Kafka or RabbitMQ, inbetween?
Filebeat -> Kafka -> Logstash Cluster -> GELF -> Graylog
Is there kinda reference architecture out there? It should be a fairly common problem and I don’t wanna reinvent the wheel.