Kafka 0.10.1.1消费者重新平衡分区的重置偏移量到最早的偏移量

时间:2018-12-07 08:29:41

标签: apache-kafka spring-integration

重新评估已分区的分区时,ConsumerCoordinator进入循环循环。

CC收到已提交的偏移量为78,但是它放弃了此偏移量,说明它已过期,然后将偏移量分配给最早的偏移量83。

Kafka群集(kafka_2.11-0.11.0.0)和消费者应用程序正在CentOS 7.4上运行。

消费应用程序配置了以下版本

org.apache.kafka:kafka_2.11:0.10.1.1
org.springframework.integration:spring-integration-kafka:2.1.0.RELEASE
org.springframework.kafka:spring-kafka:1.1.3.RELEASE

消费者日志:

[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:711] Attempt to heart beat failed for group myCGroup since member id is not valid.
[-kafka-consumer-1] o.a.k.c.c.i.ConsumerCoordinator [ConsumerCoordinator.java:333] Revoking previously assigned partitions  [my-topic-0] for group myCGroup
[-kafka-consumer-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 [AbstractMessageListenerContainer.java:242] partitions revoked:[my-topic-0]
[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:381] (Re-)joining group myCGroup
[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:390] Sending JoinGroup ({group_id=myCGroup,session_timeout=10000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0 lim=41 cap=41]}]}) to coordinator omscore4101p.prod.ch4.s.com:9092 (id: 2147483636 rack: null)
[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:406] Received successful join group response for group myCGroup: {error_code=0,generation_id=92,group_protocol=range,leader_id=consumer-1-c79a0997-0555-4f7a-abe5-59e1c33e6670,member_id=consumer-1-49dd3506-bb8c-4c87-9fa2-d27cd54488f1,members=[]}
[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:460] Sending follower SyncGroup for group fs Group to coordinator omscore4101p.prod.ch4.s.com:9092 (id: 2147483636 rack: null): {group_id=myCGroup,generation_id=92,member_id=consumer-1-49dd3506-bb8c-4c87-9fa2-d27cd54488f1,group_assignment=[]}
[-kafka-consumer-1] o.a.k.c.c.i.AbstractCoordinator [AbstractCoordinator.java:349] Successfully joined group myCGroup with generation 92
[-kafka-consumer-1] o.a.k.c.c.i.ConsumerCoordinator [ConsumerCoordinator.java:225] Setting newly assigned partitions [my-topic-0] for group myCGroup
[-kafka-consumer-1] o.a.k.c.c.i.ConsumerCoordinator [ConsumerCoordinator.java:708] Group myCGroup fetching committed offsets for partitions: [my-topic-0]
[-kafka-consumer-1] o.a.k.c.c.i.Fetcher [Fetcher.java:220] Resetting offset for partition my-topic-0 to the committed offset 78
[-kafka-consumer-1] o.a.k.c.c.i.ConsumerCoordinator [ConsumerCoordinator.java:640] Group myCGroup committed offset 78 for partition my-topic-0
[-kafka-consumer-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 [AbstractMessageListenerContainer.java:247] partitions assigned:[my-topic-0]
[-kafka-consumer-1] o.a.k.c.c.i.Fetcher [Fetcher.java:677] Discarding stale fetch response for partition my-topic-0 since its offset 83 does not match the expected offset 78
[-kafka-consumer-1] o.a.k.c.c.i.Fetcher [Fetcher.java:714] Fetch offset 78 is out of range for partition my-topic-0, resetting offset
[-kafka-consumer-1] o.a.k.c.c.i.Fetcher [Fetcher.java:340] Resetting offset for partition my-topic-0 to earliest offset.
[-kafka-consumer-1] o.a.k.c.c.i.Fetcher [Fetcher.java:583] Fetched {timestamp=-1, offset=83} for partition my-topic-0

kafka-consumer-groups的输出描述:

TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST        CLIENT-ID
my-topic  1          85              85              0          consumer-1-702fcd74-bd70-4172-b070-8c2d2ef6ca9f   /x.x.x.x    consumer-1
my-topic  0          78              83              5          consumer-1-49dd3506-bb8c-4c87-9fa2-d27cd54488f1   /x.x.x.x    consumer-1
my-topic  2          83              85              2          consumer-1-c79a0997-0555-4f7a-abe5-59e1c33e6670   /x.x.x.x    consumer-1

问题:

  • 哪些配置更改将减少此错误? “由于成员ID无效,组myCGroup尝试心跳失败。”
  • 如何指示Fetcher获取最新的偏移量?
  • 过时抵消的原因是什么?该如何解决?

0 个答案:

没有答案