Question

我在docker-compose汇集了kafka（v4.0.0）经纪人和3名动物园管理员。使用复制因子3的10个分区创建了测试主题。当创建控制台使用者没有传递--group选项（其中group.id将被自动分配）时，即使在代理被杀并且代理重新上线后，它也可以持续消费消息。

但是，如果我使用--group选项（＆＃39; console-group＆＃39;）创建一个控制台消费者，则在kafka代理被杀后会停止消息消息。

$ docker run --net=host confluentinc/cp-kafka:4.0.0 kafka-console-consumer --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --topic starcom.status --from-beginning --group console-group
<< some messages consumed >>
<< broker got killed >> 
[2017-12-31 18:34:05,344] WARN [Consumer clientId=consumer-1, groupId=console-group] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
<< no message after this >>

即使经纪人重新上线，消费者群体也不会消费任何其他消息。

奇怪的是，当我使用kafka-consumer-groups工具检查时，该消费者群体没有任何延迟。换句话说，消费者抵消正在为该消费者群体推进。没有其他消费者使用group.id运行，所以出了点问题。

根据日志，似乎该群体已经稳定下来。

kafka-2_1           | [2017-12-31 17:35:40,743] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 0 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:35:43,746] INFO [GroupCoordinator 2]: Stabilized group console-group generation 1 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:35:43,765] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 1 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:54:30,228] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 1 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:54:31,162] INFO [GroupCoordinator 2]: Stabilized group console-group generation 2 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:54:31,173] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 2 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:25,273] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 2 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:28,256] INFO [GroupCoordinator 2]: Stabilized group console-group generation 3 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:28,267] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 3 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:53,594] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 3 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:55,322] INFO [GroupCoordinator 2]: Stabilized group console-group generation 4 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 17:57:55,336] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 4 (kafka.coordinator.group.GroupCoordinator)
kafka-3_1           | [2017-12-31 18:15:07,953] INFO [GroupCoordinator 3]: Preparing to rebalance group console-group-2 with old generation 0 (__consumer_offsets-22) (kafka.coordinator.group.GroupCoordinator)
kafka-3_1           | [2017-12-31 18:15:10,987] INFO [GroupCoordinator 3]: Stabilized group console-group-2 generation 1 (__consumer_offsets-22) (kafka.coordinator.group.GroupCoordinator)
kafka-3_1           | [2017-12-31 18:15:11,044] INFO [GroupCoordinator 3]: Assignment received from leader for group console-group-2 for generation 1 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:08:59,087] INFO [GroupCoordinator 2]: Loading group metadata for console-group with generation 4 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:09:02,453] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 4 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:09:03,309] INFO [GroupCoordinator 2]: Stabilized group console-group generation 5 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:09:03,471] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 5 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:10:32,010] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 5 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:10:34,006] INFO [GroupCoordinator 2]: Stabilized group console-group generation 6 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:10:34,040] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 6 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:12:02,014] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 6 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:12:09,449] INFO [GroupCoordinator 2]: Stabilized group console-group generation 7 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:12:09,466] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 7 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:16:29,277] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 7 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:16:31,924] INFO [GroupCoordinator 2]: Stabilized group console-group generation 8 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:16:31,945] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 8 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:17:54,813] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 8 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:18:01,256] INFO [GroupCoordinator 2]: Stabilized group console-group generation 9 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:18:01,278] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 9 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:33:47,316] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 9 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:33:49,709] INFO [GroupCoordinator 2]: Stabilized group console-group generation 10 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:33:49,745] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 10 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:34:05,484] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 10 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:34:07,845] INFO [GroupCoordinator 2]: Stabilized group console-group generation 11 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 18:34:07,865] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 11 (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 19:34:16,436] INFO [GroupCoordinator 2]: Preparing to rebalance group console-group with old generation 11 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 19:34:18,221] INFO [GroupCoordinator 2]: Stabilized group console-group generation 12 (__consumer_offsets-33) (kafka.coordinator.group.GroupCoordinator)
kafka-2_1           | [2017-12-31 19:34:18,248] INFO [GroupCoordinator 2]: Assignment received from leader for group console-group for generation 12 (kafka.coordinator.group.GroupCoordinator)

主题复制正常发生。

$ docker run --net=host confluentinc/cp-kafka:4.0.0 kafka-topics --zookeeper localhost:22181 --topic starcom.status --describe
Topic:starcom.status    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: starcom.status   Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: starcom.status   Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 3    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 4    Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: starcom.status   Partition: 5    Leader: 2       Replicas: 2,1,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 6    Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: starcom.status   Partition: 7    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 8    Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 9    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1

$ docker run --net=host confluentinc/cp-kafka:4.0.0 kafka-topics --zookeeper localhost:22181 --topic starcom.status --describe
Topic:starcom.status    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: starcom.status   Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 2,3
        Topic: starcom.status   Partition: 1    Leader: 2       Replicas: 1,2,3 Isr: 3,2
        Topic: starcom.status   Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 3,2
        Topic: starcom.status   Partition: 3    Leader: 3       Replicas: 3,2,1 Isr: 3,2
        Topic: starcom.status   Partition: 4    Leader: 3       Replicas: 1,3,2 Isr: 3,2
        Topic: starcom.status   Partition: 5    Leader: 2       Replicas: 2,1,3 Isr: 3,2
        Topic: starcom.status   Partition: 6    Leader: 3       Replicas: 3,1,2 Isr: 2,3
        Topic: starcom.status   Partition: 7    Leader: 2       Replicas: 1,2,3 Isr: 3,2
        Topic: starcom.status   Partition: 8    Leader: 2       Replicas: 2,3,1 Isr: 3,2
        Topic: starcom.status   Partition: 9    Leader: 3       Replicas: 3,2,1 Isr: 3,2

$ docker run --net=host confluentinc/cp-kafka:4.0.0 kafka-topics --zookeeper localhost:22181 --topic starcom.status --describe
Topic:starcom.status    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: starcom.status   Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: starcom.status   Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 3    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 4    Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: starcom.status   Partition: 5    Leader: 2       Replicas: 2,1,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 6    Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: starcom.status   Partition: 7    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: starcom.status   Partition: 8    Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: starcom.status   Partition: 9    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1

这是（融合的）kafka控制台消费者的限制吗？基本上，我试图通过运行这个较小的测试来确保我真正的Java Kafka消费者能够在代理停机时间内存活下来。

任何帮助将不胜感激。

编辑（2018年！）：

我完全重新创建了我的docker（-compose）环境，并且能够重现这一点。这一次我创建了＆＃39; new-group＆＃39;代理重新启动后，消费者组和控制台消费者抛出了错误。从那时起，消息就不会消耗掉。同样，根据消费者群体工具，消费者抵消正在向前发展。

[2018-01-01 19:18:32,935] ERROR [Consumer clientId=consumer-1, groupId=new-group] Offset commit failed on partition starcom.status-4 at offset 0: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-01-01 19:18:32,936] WARN [Consumer clientId=consumer-1, groupId=new-group] Asynchronous auto-commit of offsets {starcom.status-4=OffsetAndMetadata{offset=0, metadata=''}, starcom.status-5=OffsetAndMetadata{offset=0, metadata=''}, starcom.status-6=OffsetAndMetadata{offset=2, metadata=''}} failed: Offset commit failed with a retriable exception. You should retry committing offsets. The underlying error was: This is not the correct coordinator. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Answer 1

原来是一个码头新手错误。

当我在kafka-console-consumer shell上编辑 ctrl + c 时，容器（group.id：＆＃34; console-group＆＃34;）是进入分离模式。在我运行docker ps [-n | -a]命令之前，我一直都不知道。当我使用相同的命令（docker run --net=host confluentinc/cp-kafka:4.0.0 kafka-console-consumer --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --topic starcom.status --from-beginning --group console-group）启动另一个控制台消费者时，消费者加入了相同的＆＃34;控制台组＆＃34;。这就是为什么后续消息（显然我是使用相同的分区键生成消息）被第一个在后台运行的消费者所消耗，并给我错误的印象，即消息丢失了。这就是为什么消费者群体命令显示正确的偏移量提升的原因。在不同窗口中将原始使用者重新附加到前台（docker attach <<container-id>>）之后，现在我看到根据分区分配在两个不同的控制台中使用所有生成的消息。一切都按预期工作。对于虚假警报感到抱歉，但希望遇到同样问题的人能从中得到一些暗示。

Answer 2

总结一下，如果我想消费一些消息，那么在docker环境中设置kafka-console-consumer的正确方法应该是

docker run --net=host --rm -i -t \ 
    confluentinc/cp-kafka:4.0.0 \
      kafka-console-consumer --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --topic foo.bar

注意有--rm，-i，-t选项。＆＃39; -i和-t＆＃39;如果您传递了--max-messages，则不需要在这种情况下控制台将正常退出并停止并拆除容器。

在docker容器中运行的kafka-console-consumer停止使用消息

2 个答案: