Adding Graylog to already existing ELK stack

(Matej Zerovnik) #1

Hello,

we already have ELK stack set up and in use, but we are missing some of the nice features Graylog is offering, mainly alerting and nice GUI, but also retentions (which can be done via CLI, but it’s not as nice as gui) and streams (although we do basic streams in Logstash).

Going Graylog only is probably not an option, because we do quite some log manipulation & enrichment in Logstash and would like to keep Kibana for it’s analytics capabilities.

If we want to forward logs from Logstash to Graylog, what would be the best option? As far as I checked, most of people use one of GELF / TCP / HTTP / Kafka, but how do those plugins perform when Graylog is not accessible. Are logs dropped or saved in queue and sent when graylog is back online? I would like to keep the system simple, but still have ensured delivery.

If I want to have encrypted logs, GELF is probably out of the game?

Which fields does Graylog need in the logs input for proper working? Which fields are necessary?
What field does Graylog use for timestamp? Can it use @timestamp?

Is Logstash’s output-gelf plugin supported by Graylog or community?

What else do I have to take into account if I want to put Graylog between Logstash and ES?

Thanks for help, Matej

(Jan Doberstein) #2

Hej @matejzero

why not using the beats input on Graylog and the native beats protocol to transfer messages from logstash to Graylog?

To get an idea what is supported by whom please read this posting.

regards
Jan

(Matej Zerovnik) #3

This was actually my first idea, but Logstash doesn’t seem to have a beats output plugin, only input plugin, so that’s why I started looking at the alternatives.

(Jan Doberstein) #4

you might want to watch: https://github.com/elastic/logstash/issues/5867

sorry that I didn’t check if that is possible. If you need some buffers, you might want to use GELF over Kafka or AMQP (depending on your needs).

(Matej Zerovnik) #5

Ooo I see there is some movement toward beats output plugin, thanks for the link. I need to wait for Graylog 3.0, since we have ES 5.0 cluster and by that time, plugin might be ready:)

In the worst case, I’ll go with Kafka, although I would like to keep my architecture simple.

I want beats or kafka to have backpressure control (or at least a buffer) and SSL encryption of the data, since some of it is send over WAN.

Thanks for help.

(Tom) #6

Hi Matej,

Did you find a solution for this? We’re facing the same issue. We’re using Security Onion w/ELK and want logstash to forward logs/alerts to Graylog.

Thanks,

Tom