Kafka input without Zookeeper connection

[neuwerld (Andreas)]

I have been testing around to get an input to get data from Kafka. First i tried with the “old” version of the Kafka plugin which is version 0.9.x something. This did not work and only gave me the following logs:

2021-09-16T13:52:08.624+02:00 ERROR [KafkaTransport] WakeupException in poll. Kafka server is not responding.
2021-09-16T13:52:08.627+02:00 ERROR [KafkaTransport] Caught recoverable exception. Retrying
2021-09-16T13:52:08.666+02:00 ERROR [KafkaTransport] WakeupException in poll. Kafka server is not responding.
2021-09-16T13:52:08.666+02:00 ERROR [KafkaTransport] Caught recoverable exception. Retrying
2021-09-16T13:52:12.745+02:00 ERROR [KafkaTransport] WakeupException in poll. Kafka server is not responding.
2021-09-16T13:52:12.746+02:00 ERROR [KafkaTransport] Caught recoverable exception. Retrying

I then found somebody who updated this plugin to a newer version:

GitHub - ceyhunn/graylog-kafka-plugin

With this version you cannot skip the Zookeeper configuration, our zookeeper nodes cannot be accessed from where they are setup so I can’t get a connection to Zookeeper.

I wonder why you can’t disable the Zookeeper part in the new version of the plugin and if somebody has any input or solution to my problem?

Best regards,
Andreas

[aaronsachs]

What version of Graylog are you running? IIRC we made improvements around Kafka versions in > 4.0, so this plugin shouldn’t be necessary.

[neuwerld (Andreas)]

We are using version 3.3.9, have read that it should be better in newer versions. Do you know if it is possible to use the Kafka plugin without using Zookeeper connection?