Hello
I have this error as a popup in my browser
http://ip:9000/api/system/fields HTTP 500
{“message”:“Couldn’t read cluster state for indices graylog_*”,“details”:}
There are not exceptions in kafka, elasticserch, mongo or zookeeper.
But when i try to consume some kafka topic i have the exception in Graylog container logs
kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {“listener_security_protocol_map”:{“LISTENER_DOCKER_INTERNAL”:“PLAINTEXT”,“LISTENER_DOCKER_EXTERNAL”:“PLAINTEXT”},“endpoints”:[“LISTENER_DOCKER_INTERNAL://kafka:19092”,“LISTENER_DOCKER_EXTERNAL://ip:9092”],“jmx_port”:-1,“host”:“kafka”,“timestamp”:“1555925480367”,“port”:19092,“version”:4}
at kafka.cluster.Broker$.createBroker(Broker.scala:85) ~[graylog.jar:?]
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:563) ~[graylog.jar:?]
at kafka.utils.ZkUtils$$anonfun$getCluster$1.apply(ZkUtils.scala:561) ~[graylog.jar:?]
at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[graylog.jar:?]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[graylog.jar:?]
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[graylog.jar:?]
at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[graylog.jar:?]
at kafka.utils.ZkUtils.getCluster(ZkUtils.scala:561) ~[graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:635) [graylog.jar:?]
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:627) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:627) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:627) [graylog.jar:?]
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:626) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:1001) [graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:163) [graylog.jar:?]
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) [graylog.jar:?]
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:105) [graylog.jar:?]
at org.graylog2.inputs.transports.KafkaTransport.doLaunch(KafkaTransport.java:207) [graylog.jar:?]
at org.graylog2.plugin.inputs.transports.ThrottleableTransport.launch(ThrottleableTransport.java:75) [graylog.jar:?]
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) [graylog.jar:?]
at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.kafka.common.protocol.SecurityProtocol.LISTENER_DOCKER_INTERNAL
at java.lang.Enum.valueOf(Enum.java:238) ~[?:1.8.0_181]
at org.apache.kafka.common.protocol.SecurityProtocol.valueOf(SecurityProtocol.java:27) ~[graylog.jar:?]
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:48) ~[graylog.jar:?]
at kafka.cluster.Broker$$anonfun$1.apply(Broker.scala:73) ~[graylog.jar:?]
at kafka.cluster.Broker$$anonfun$1.apply(Broker.scala:72) ~[graylog.jar:?]
at scala.collection.immutable.List.map(List.scala:273) ~[graylog.jar:?]
at kafka.cluster.Broker$.createBroker(Broker.scala:72) ~[graylog.jar:?]
… 29 more
2019-04-22 12:56:28,724 INFO : kafka.consumer.ZookeeperConsumerConnector - [graylog2_a83edb9d3f05-1555937782679-2b101f80], end rebalancing consumer graylog2_a83edb9d3f05-1555937782679-2b101f80 try #3
2019-04-22 12:56:28,724 INFO : kafka.consumer.ZookeeperConsumerConnector - [graylog2_a83edb9d3f05-1555937782679-2b101f80], Rebalancing attempt failed. Clearing the cache before the next rebalancing operation is triggered
2019-04-22 12:56:28,724 INFO : kafka.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1555937782683] Stopping leader finder thread
2019-04-22 12:56:28,724 INFO : kafka.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1555937782683] Stopping all fetchers
2019-04-22 12:56:28,724 INFO : kafka.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1555937782683] All connections stopped
2019-04-22 12:56:28,724 INFO : kafka.consumer.ZookeeperConsumerConnector - [graylog2_a83edb9d3f05-1555937782679-2b101f80], Cleared all relevant queues for this fetcher
2019-04-22 12:56:28,724 INFO : kafka.consumer.ZookeeperConsumerConnector - [graylog2_a83edb9d3f05-1555937782679-2b101f80], Cleared the data chunks in all the consumer message iterators
2019-04-22 12:56:28,724 INFO : kafka.consumer.ZookeeperConsumerConnector - [graylog2_a83edb9d3f05-1555937782679-2b101f80], Committing all offsets after clearing the fetcher queues
2019-04-22 12:56:30,725 ERROR: org.graylog2.shared.inputs.InputLauncher - The [org.graylog2.inputs.raw.kafka.RawKafkaInput] input with ID <5cbd776e9194be000bd2ca32> misfired. Reason: graylog2_a83edb9d3f05-1555937782679-2b101f80 can’t rebalance after 4 retries.
org.graylog2.plugin.inputs.MisfireException: kafka.common.ConsumerRebalanceFailedException: graylog2_a83edb9d3f05-1555937782679-2b101f80 can’t rebalance after 4 retries
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: kafka.common.ConsumerRebalanceFailedException: graylog2_a83edb9d3f05-1555937782679-2b101f80 can’t rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660) ~[graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967) ~[graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:1001) ~[graylog.jar:?]
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:163) ~[graylog.jar:?]
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:101) ~[graylog.jar:?]
at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:105) ~[graylog.jar:?]
at org.graylog2.inputs.transports.KafkaTransport.doLaunch(KafkaTransport.java:207) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.transports.ThrottleableTransport.launch(ThrottleableTransport.java:75) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
… 7 more
2019-04-22 12:56:30,727 INFO : org.graylog2.inputs.InputStateListener - Input [Raw/Plaintext Kafka/5cbd776e9194be000bd2ca32] is now FAILED
2019-04-22 12:56:40,211 ERROR: org.graylog2.indexer.cluster.Cluster - Couldn’t read cluster health for indices [graylog_*] (elasticsearch)
2019-04-22 12:56:40,211 INFO : org.graylog2.periodical.IndexerClusterCheckerThread - Indexer not fully initialized yet. Skipping periodic cluster check.