Question

我正在尝试将KStream与KTable结合在一起。没有联接，从中间主题“按ID分配book-attribute-by-id”中读取内容就毫无问题。

KTable的示例味精：

{key: {id: 1}
 value: {id: 1, attribute_name: "weight"}}

KStream的示例味精：

{key: {id: 1},
 value: {id: 1, book_id: 1, attribute_id: 1, value: 200}}

“最终汇总”主题所需的输出：

{key: {id: 1}, 
 value: {book_id: 1, attribute_name: "weight", value: 200}}
{key: {id: 1}, 
 value: {book_id: 1, attribute_name: "number_of_pages", value: 450}}

这是代码

    KStream<DefaultId, BookAttribute> bookAttributeStream = builder.stream(bookAttributeTopic, Consumed.with(defaultIdSerde, bookAttributeSerde));
    KStream<DefaultId, BookValueInt> bookValueIntStream = builder.stream(bookValueIntTopic, Consumed.with(defaultIdSerde, bookValueIntSerde));

    bookAttributeStream
        .selectKey((k, v) -> k.getId())
        .to("book-attribute-by-id", Produced.with(Serdes.Integer(), bookAttributeSerde));

    KTable<Integer, BookAttribute> bookAttributeByIdTable = builder.table("book-attribute-by-id", Consumed.with(Serdes.Integer(), bookAttributeSerde));

    // when the snippet below is commented out, consuming "book-attribute-by-id" works. 
    bookValueIntStream
        .selectKey((k, v) -> v.getAttribute_id())
        .join(bookAttributeByIdTable, (intValue, attribute) -> {
                System.out.println("intValue: " + intValue);
                System.out.println("attribute: " + attribute);
                return new BookAttributeValue(intValue, attribute);
            });

加入KStream和KTable时发生异常：

线程“ xxx-StreamThread-1”中的异常 org.apache.kafka.streams.errors.TopologyBuilderException：无效拓扑构建：流线程[xxx-StreamThread-1]找不到主题：依ID的书籍属性 org.apache.kafka.streams.processor.internals.StreamPartitionAssignor $ CopartitionedTopicsValidator.validate（StreamPartitionAssignor.java:792）

Answer 1

我认为您正在使用kafka-streams 1.0.0

问题在于您必须为流创建输入主题。

在您的情况下，主题是：using System; using System.Data.SqlClient; namespace Contoso { public static class UsersRepository { private static string ConnectionString = @"Data Source=(local); Database=Users;User Id=sa;Password=password;"; public static User Load(int userId) { User user = new User(); SqlConnection connection = new SqlConnection(ConnectionString); connection.Open(); SqlCommand command = new SqlCommand("SELECT * FROM Users WHERE UserId = " + userId, connection); var reader = command.ExecuteReader(); while (reader.Read()) { user.Name = reader["Name"].ToString(); user.DateOfBirth = DateTime.Parse(reader["DateOfBirth"].ToString()); user.Country = reader["Country"].ToString(); } connection.Close(); return user; } } }和那些变量值：book-attribute-by-id，bookAttributeTopic。

对于加入，Kafka Streams必须确保加入主题中的分区数相等。尝试获取主题bookValueIntTopic的元数据时，抛出异常。

在运行应用程序之前，您必须手动创建book-attribute-by-id主题

在较新版本的kafka-streams中，在验证分区数量之前先检查主题是否存在。

从中间主题将KStream与KTable结合会导致异常

1 个答案: