I have a Kafka instance on my localhost and I’m trying to create an input in grayling to pull the messages from Kafka. I have tested the Kafka service alone, and it works fine.
Setup info:
The zookeeper is running in localhost:2181
The Kafka is running in localhost:9092
The Graylog also running in the same machine.
I have went through all the default input creation setup in graylog for Kafka and no luck. Anyone have any setup guide to create input type Kafka (GELF, sys or plan text)?
If you run the consumer offset in Kafka, do you see the consumer show up? I believe the group is graylog2.
/path/to/kafka/bin/kafka-consumer-groups --zookeeper localhost:2181 --describe --group graylog2
OR (deprecated but still might work)
/path/to/kafka/bin/kafka-consumer-offset-checker --group graylog2 --topic TOPIC_NAME --zookeeper localhost:2181
Do you see any bytes showing on the input? If it was stored in GELF format, I would use GELF and if syslog sent it, I would use Syslog Kafka. For everything else, I would use Raw/Plain Text Kafka.
Thank you. I used Raw/Plain Text Kafka and it works 