Question

因此，我正在使用 kafka connect，kafka流和elasticsearch 构建CDC管道。我对这三项技术都很陌生。因此，我正在使用 Kafka connect从mysql数据库获取数据，并根据商品类型创建主题（效果很好）。这是2件商品，但最终，我将有多种商品类型。

例如。卡夫卡主题创建

test.catalog.catalog_air_cooler
test.catalog.catalog_air_cooler_eav
test.catalog.catalog_air_purifier
test.catalog.catalog_air_purifier_eav。

我的最终目标是构建所有主题的汇总视图，并将其作为索引存储在弹性搜索中。我正在使用 kafka流。

private void constructReport() {
        Properties props = kafkaConfig.populateKafkConfigMap("DwCatrgoryReport");
        List<String> itemTypes = new ArrayList<String>();
// the item types for which topics are created
        itemTypes.add("air_cooler");
        itemTypes.add("air_purifier");
        for (int i = 0; i < itemTypes.size(); i++) {
            String itemType = itemTypes.get(i);
            // I am dynamically constructing topic names.
            String topicName = "test.catalog.catalog_"+itemType;
            StreamsBuilder stremBuilder = new StreamsBuilder();
           //am constructing different streams for different topics
            KTable<String, String> catalogStream = stremBuilder.table(topicName);
            KTable<String, String> catalogEavStream = stremBuilder.table(topicName+"_eav");

           //changing the key of table for join.(omitting code)

            //joining catalog and catlogeav table
            KStream<String, String> stream = catalogStream.join(catalogEavStream, (leftValue, rightValue) -> kafkaStreamsListenerOperationsHelperService
                    .createAggregateStream(leftValue, rightValue, ReqType.CATALOG)).toStream();
            callEsAndCreateIndex(stream, ElasticSearchTopics.DW_CATEGORY_GROUP_TOPIC);
            final KafkaStreams streams = new KafkaStreams(stremBuilder.build(), props);
                streams.start();
                Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
        }

在上面的代码中，我正在创建一个流，将它们连接起来并存储在elasticsearch中。

那么发生的是，在弹性搜索中为air_cooler聚合（两个表）创建了索引，但没有为air_purifier创建索引。我收到以下异常。

java.lang.IllegalStateException: Consumer was assigned partitions [ktm.catalog.catalog_air_cooler_eav-0] which didn't correspond to subscription request [ktm.catalog.catalog_air_purifier, ktm.catalog.catalog_air_purifier_eav, DwCatrgoryReport-KTABLE-AGGREGATE-STATE-STORE-0000000012-repartition, DwCatrgoryReport-KTABLE-AGGREGATE-STATE-STORE-0000000007-repartition]
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.handleAssignmentMismatch(ConsumerCoordinator.java:218)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:264)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:424)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:963)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:859)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:819)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:788)

在哪里可能解决此异常方面需要帮助或指导。预先感谢。

在为Kafka流中的不同主题创建聚合时发生IlegalStateException

0 个答案: