Kafka Streams:Kafka Streams应用程序重新平衡

时间:2018-07-02 17:25:57

标签: apache-kafka apache-kafka-streams kafka-streams-scala

在所有Kafka经纪人重新启动以升级offset.retention.minutes设置(将其增加到60天)后,使用该卡夫卡Streams应用程序的用户被卡住,并且消费组显示了重新平衡:

bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort 

Warning: Consumer group 'stream-processor' is rebalancing.

TOPIC                  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
customers              0          84263           84288           25              -               -               -
customers              1          71731           85068           13337           -               -               -
customers              10         71841           84801           12960           -               -               -
customers              11         84273           84336           63              -               -               -
customers              12         84189           84297           108             -               -               -
customers              13         83969           84051           82              -               -               -
customers              14         84693           84767           74              -               -               -
customers              15         84472           84556           84              -               -               -
customers              2          84175           84239           64              -               -               -
customers              3          71719           71719           0               -               -               -
customers              4          71446           84499           13053           -               -               -
customers              5          84291           84361           70              -               -               -
customers              6          71700           71700           0               -               -               -
customers              7          72003           85235           13232           -               -               -
customers              8          84521           84587           66              -               -               -
customers              9          71513           71513           0               -               -               -
customers-intermediate 0          102774          102792          18              -               -               -
customers-intermediate 1          102931          103028          97              -               -               -
customers-intermediate 10         102883          102965          82              -               -               -
customers-intermediate 11         102723          102861          138             -               -               -
customers-intermediate 12         102797          102931          134             -               -               -
customers-intermediate 13         102339          102460          121             -               -               -
customers-intermediate 14         103227          103321          94              -               -               -
customers-intermediate 15         103231          103366          135             -               -               -
customers-intermediate 2          102637          102723          86              -               -               -
customers-intermediate 3          84620           103297          18677           -               -               -
customers-intermediate 4          102596          102687          91              -               -               -
customers-intermediate 5          102980          103071          91              -               -               -
customers-intermediate 6          84385           103058          18673           -               -               -
customers-intermediate 7          103559          103652          93              -               -               -
customers-intermediate 8          103243          103312          69              -               -               -
customers-intermediate 9          84211           102772          18561           -               -               -
events                 15         11958555        15231834        3273279         -               -               -
events                 3          1393386         16534651        15141265        -               -               -
events                 4          1149540         15390069        14240529        -               -               -
visitors               15         2774874         2778873         3999            -               -               -
visitors               3          603242          603324          82              -               -               -
visitors               4          565266          565834          568   

流媒体应用程序也重新启动,之后我可以看到一些处理日志大约20小时,然后停止处理。

已经有两天了。但也值得一提的是,您在上面看到的所有主题都有16个分区,但有些显示其中三个分区(访问者,事件)。但是,我可以描述这些主题,并且它们的分区像往常一样分布良好,在那里我找不到任何奇怪的地方。

可能会发生什么? 重新启动应用程序后,我可以再次看到所有分区,并且应用程序正在使用主题分区。但是,许多(大多数)分区丢失了偏移量。自从我更改了offset.retention.minutes设置以来,就不会发生这种情况。

events                 0          -               14538261        -               stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
events                 1          49070           13276094        13227024        stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 10         -               15593746        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 11         -               15525487        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 12         -               21863908        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 13         -               15810925        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 14         -               13509742        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events                 15         11958555        15231834        3273279         stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
...

Kafka 1.1.0,Kafka Streams 1.0

0 个答案:

没有答案
相关问题