Raw/Plaintext Kafka input failing

Graylog Central (peer support)
1

I’m trying to connect graylog to the AWS kafka MKS and getting this error:

Input xxxxxxxxxxxxxxx has failed to start on node xxxxxxxxxxxxxxxxxxxxxx for this reason: »aws_kafka_001_ip-xxxxxxxxxxxxxxxx can’t rebalance after 4 retries.«. This means that you are unable to receive any messages from this input. This is mostly an indication for a misconfiguration or an error.

Any ideas why this could be happening ?

2

Hello,

Judging from your error message something seams not to be configured correctly.
Is it possible to show any of your configuration?
Maybe the version of Graylog server or any other environment variables that you have setup?

3

Hello, @gsmith thank you for replying. This is the configuration in graylog:

Graylog - Inputs
Graylog - Inputs532×915 52.3 KB

The graylog version is 3.1

4

Hello,

After Googling I came across this.

rebalance.backoff.ms defines the time for which Kafka will wait before rebalancing & zookeeper.session.timeout.ms parameter specifies the time for which Kafka will wait to connect to zookeeper. So, try adjusting these two parameters.

See if that helps

5

Hello @gsmith . I came across that too and made the change but that did not fix the issue.

6

Out of curiosity have you tried different port for the kafka input and/or possible with a different Input?
Do you see anything in your journal that may pertain to tis issue?

7

I have another Kafka input running on this graylog but that Kafka is not the MKS (aws kafka) and it is working as expected. So pretty much the only difference between the 2 inputs is that one is getting data from a Kafka cluster that is maintained by me and the input failing is the one that I’m trying to configure to get data from MKS. Not sure if there is a compatibility issue or something I don’t know about.

8

Sometime this can be the cause.

9

You mean changing the zookeeper port to something other than 2181 ? I’ll try that and let you know if that fixes the issue. Thank you

1 Like
10

Hello @gsmith unfortunately the port cannot be changed in MKS.

11

Is it possible to show a logical diagram on what your trying top accomplish? What I get of all this is that you created a two Inputs that are the same. One INPUT is running fine and the other INPUT one is not.
By chance have you tried to use a different type of Input. To be honest I’m not sure what Input you are using.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.