Question

I want to ReKey a GlobalKTable (probably while initializing it, as I believe they are read only once created).

Is this possible?

I have two topics that I am working with in a Spring/Java Kafka Streams app. The first is not compacted, the second is. Both use Avro for their keys and values.

The app streams records from the first (non-compacted) topic, and attaches additional data from the compacted topic via KStream#leftJoin. The compacted topic has been brought into the app as a GlobalKTable, created via StreamsBuilder#globalTable() and needs to stay this way (I need every record from all partitions of the topic available in each instance of the app).

I know there is talk of supporting non primary key joins (https://issues.apache.org/jira/browse/KAFKA-3705), but to my knowledge, I can't do this yet...

@Configuration
@EnableKafkaStreams
public class StreamsConfig {

  @Autowired
  private MyCustomSerdes serdes;

  @Bean
  public KStream<AvroKeyOne, AvroValueOne> reKeyJoin(StreamsBuilder streamsBuilder) {

    GlobalKTable<AvroKeyOne, AvroValueOne> globalTable = streamsBuilder.globalTable("topicOne", Consumed.with(
      serdes.getAvroKeyOne()
      serdes.getAvroValueOne()
    ));

    KStream<AvroKeyTwo, AvroValueOne> kStream = streamsBuilder.stream("topicTwo", Consumed.with(
      serdes.getAvroKeyTwo(),
      serdes.getAvroValueOne()
    ));

    kStream.join(
      globalTable,
       /**
        * the KeyValueMapper. I need to rekey the Global table as well to the
        * corresponding String (which it's data will have) if I want this join
        * to return results
        */
      (streamKey, streamValue) -> {return streamKey.getNewStringKey()},
      (/**ValueJoiner Deal**/)
    );
  }

}

Answer 1

我想重新设置GlobalKTable的密钥（可能是在初始化它时，因为我相信它们一旦创建就只能读取）。

这可能吗？

今天没有对此的直接支持。您已经提到了即将进行的工作，例如添加support to global tables for non-primary-key joins，但这还不可用。

您今天可以做的事情：您可以将原始Kafka主题重新设置（重新分区）为新主题，然后将重新设置主题的内容读取到全局KTable中。也许这是您的选择。

Is it possible to ReKey a GlobalKTable?

1 个答案: