So I’ve really been having a hard time grasping how to properly monitor Kafka inputs. Based on the configuration it is connecting to zookeeper, but in kafka I am not seeing a consumer group all the nodes join, so I’m not sure how the different graylog nodes are managing their offsets.
It does look like you need a 1:1 kafka partition to graylog input thread match, which makes sense.
Are there any best practices or ways people have found to actually manage the kafka consumer offset(consumer lag in processing the queue) for the modern versions of kafka?
The question really is: “How far behind is graylog in processing the kafka queue?”