We have AWS MSK kafka cluster with 1.1.1 aiming to receive messages on graylog 3.1 deployed on kubernetes. I have been reading. Since graylog uses kafka client to communicate with kafka there is some version incompatability. Right now. If zookeeper host and port is set with plaintext kafka input. Exception is thrown. Any suggestions is highly appreciated.
2020-07-16 14:50:49,518 INFO [ZookeeperConsumerConnector] - [graylog2_graylogingress-0.graylogingress.graylog-ingress.svc.cluster.local-1594902241312-41b3587e], exception during rebalance - {} kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"listener_security_protocol_map":{"CLIENT":"PLAINTEXT","CLIENT_SECURE":"SSL","REPLICATION":"PLAINTEXT","REPLICATION_SECURE":"SSL"},"endpoints":["CLIENT://b-1.tooling-stack-kafka.pg6kf7.c3.kafka.eu-central-1.amazonaws.com:9092","CLIENT_SECURE://b-1.tooling-stack-kafka.pg6kf7.c3.kafka.eu-central-1.amazonaws.com:9094","REPLICATION://b-1-internal.tooling-stack-kafka.pg6kf7.c3.kafka.eu-central-1.amazonaws.com:9093","REPLICATION_SECURE://b-1-internal.tooling-stack-kafka.pg6kf7.c3.kafka.eu-central-1.amazonaws.com:9095"],"rack":"euc1-az3","jmx_port":9099,"host":"b-1.tooling-stack-kafka.pg6kf7.c3.kafka.eu-central-1.amazonaws.com","timestamp":"1594866945238","port":9092,"version":4} at kafka.cluster.Broker$.createBroker(Broker.scala:85) ~[graylog.jar:?] at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:563) ~[graylog.jar:?] at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:561) ~[graylog.jar:?] at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[graylog.jar:?] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[graylog.jar:?] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[graylog.jar:?] at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[graylog.jar:?] at kafka.utils.ZkUtils.getCluster(ZkUtils.scala:561) ~[graylog.jar:?]