we already have ELK stack set up and in use, but we are missing some of the nice features Graylog is offering, mainly alerting and nice GUI, but also retentions (which can be done via CLI, but it’s not as nice as gui) and streams (although we do basic streams in Logstash).
Going Graylog only is probably not an option, because we do quite some log manipulation & enrichment in Logstash and would like to keep Kibana for it’s analytics capabilities.
If we want to forward logs from Logstash to Graylog, what would be the best option? As far as I checked, most of people use one of GELF / TCP / HTTP / Kafka, but how do those plugins perform when Graylog is not accessible. Are logs dropped or saved in queue and sent when graylog is back online? I would like to keep the system simple, but still have ensured delivery.
If I want to have encrypted logs, GELF is probably out of the game?
Which fields does Graylog need in the logs input for proper working? Which fields are necessary?
What field does Graylog use for timestamp? Can it use @timestamp?
Is Logstash’s output-gelf plugin supported by Graylog or community?
What else do I have to take into account if I want to put Graylog between Logstash and ES?
This was actually my first idea, but Logstash doesn’t seem to have a beats output plugin, only input plugin, so that’s why I started looking at the alternatives.
Ooo I see there is some movement toward beats output plugin, thanks for the link. I need to wait for Graylog 3.0, since we have ES 5.0 cluster and by that time, plugin might be ready:)
In the worst case, I’ll go with Kafka, although I would like to keep my architecture simple.
I want beats or kafka to have backpressure control (or at least a buffer) and SSL encryption of the data, since some of it is send over WAN.
