Sending syslog via KAFKA into Graylog
@jalogisch
View on Github
Open Issues
Stargazers
If your setup needs to buffer log messages during the transport to Graylog or Graylog is not accessible from all network segments, you can use Apache Kafka as a message broker from which Graylog will pull messages, once they are available.
Please be aware that Graylog will connect to Apache ZooKeeper and fetch the topics defined by the configured regular expressing. Adding SSL/TLS or authentification information is not possible with the latest stable version of Graylog (2.1.0 at the time of writing).
NOTE: This Guide will not give you a complete copy & paste howto, but it will guide you through the setup process and provide additional information if necessary.
@Please do not follow the described steps blindly if you don’t know how to deal with common issues yourself.
In the scenario used in this guide, a syslog message will run through the following stages:
- Message sent from rsyslog to Logstash via TCP or UDP
- Message sent from Logstash to Apache Kafka
- Message pulled and consumed from Apache Kafka by Graylog (via Kafka input)
- Structured syslog information extracted from JSON payload by Graylog
If you run rsyslog 8.7.0 or higher with support for Apache Kafka, the message can run through the following stages:
- Message sent from rsyslog to Apache Kafka
- Message pulled and consumed from Apache Kafka by Graylog (via Kafka input)
- Structured syslog information extracted from JSON payload by Graylog
We assume that there is an Apache Kafka instance running on kafka.int.example.org (192.168.100.10) and a Graylog instance is running on graylog.int.example.org (192.168.1.10). Additionally, the logs will be generated by Linux systems syslog.o1.example.org (192.168.50.30) and syslog.o2.example.org (192.168.2.30).
All Systems are running Ubuntu Linux , so you might need to adjust some configuration path settings on different operating systems.