Question

我使用Spring Kafka API实现Kafka使用者手动偏移管理： @KafkaListener（topics =“some_topic”） public void onMessage（@Payload Message message，Acknowledgement acknowledgement）{ if（someCondition）{ acknowledgment.acknowledge（）; } } 在这里，我希望消费者只有在someCondition成立时才提交偏移量。否则，消费者应该睡一段时间并再次阅读相同的消息。卡夫卡配置： @豆 public ConcurrentKafkaListenerContainerFactory＆lt; String，String＆gt; kafkaListenerContainerFactory（）{ ConcurrentKafkaListenerContainerFactory＆lt; String，String＆gt; factory = new ConcurrentKafkaListenerContainerFactory＆lt;＆gt;（）; factory.setConsumerFactory（new DefaultKafkaConsumerFactory＆lt;＆gt;（consumerConfig（））; 。factory.getContainerProperties（）setAckMode（MANUAL）; 返厂; } 私有地图＆lt; String，Object＆gt; consumerConfig（）{ Map＆lt; String，Object＆gt; props = new HashMap＆lt;＆gt;（）; ... props.put（ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG，“false”）; ... 回归道具; } 使用当前配置，如果someCondition == false，则consumer不提交偏移量，但仍会读取下一条消息。如果没有执行Kafka确认，是否有办法让消费者重读消息？

Answer 1

正如@Gary已经指出的那样，你的方向是正确的，public class Receiver implements AcknowledgingMessageListener<Integer, String>, ConsumerSeekAware { private ConsumerSeekCallback consumerSeekCallback; @Override public void onMessage(ConsumerRecord<Integer, String> record, Acknowledgment acknowledgment) { if (/*some condition*/) { //process acknowledgment.acknowledge(); //send ack } else { consumerSeekCallback.seek("your.topic", record.partition(), record.offset()); } } @Override public void registerSeekCallback(ConsumerSeekCallback consumerSeekCallback) { this.consumerSeekCallback = consumerSeekCallback; } @Override public void onPartitionsAssigned(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) { // nothing is needed here for this program } @Override public void onIdleContainer(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) { // nothing is needed here for this program } }是这样做的方法。当我遇到这个问题时，我今天无法找到它的代码示例。以下是任何想要解决问题的人的代码。

\(([^)]+)\)\s*[<\\[].+?[>\\]]

Answer 2

您可以停止并重新启动容器，然后重新发送。

即将发布的1.1版本，您可以seek to the required offset并重新发布。

但是如果已经检索过后面的消息，你仍会先看到它们，所以你也必须放弃这些消息。

second milestone有此功能，我们预计会在下周发布。

Answer 3

您可以尝试使用nack(long sleep)（其中唯一参数代表sleep interval ms）来实现上述功能。

来自Spring for Apache Kafka documentation：

从 2.3版开始，确认界面有两个其他方法nack（长时间睡眠）和nack（int索引，长时间睡眠）。第一个与记录监听器一起使用，第二个与批处理一起使用听众。为您的侦听器类型调用错误的方法将引发一个IllegalStateException。

将以上信息应用到代码示例中，我们得到：

@Component
@Slf4j
public class ExampleConsumer {
    private boolean nonError = false;
    
    @KafkaListener(topics = "topic_name")
    private void consumeSelectingMsgFromMailbox(ConsumerRecord<String, KafkaEventPojo> record, Acknowledgment ack) {
        log.info("Received record topic:{} partition:{} offset:{}", record.topic(), record.partition(), record.offset());
        
        if (nonError) {
            log.info("ACK: {}", offset);
            ack.acknowledge(); //send ack
            if (offset % 2 == 0)
                nonError = false;
        } else {
            ack.nack(0); // immediate seek - no sleep time for consumer
            nonError = true;
        }
    }
}

配置如下：

@Configuration
@EnableKafka
public class KafkaConsumerConfig {
    private ConcurrentKafkaListenerContainerFactory<String, KafkaEventPojo> factory;

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        // ...
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        // ...

        return props;
    }

    @Bean
    public ConsumerFactory<String, KafkaEventPojo> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, KafkaEventPojo> kafkaListenerContainerFactory() {
        if (this.factory == null) {
            this.factory =
                    new ConcurrentKafkaListenerContainerFactory<>();
            factory.setConsumerFactory(consumerFactory());
            factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
        }

        return this.factory;
    }

该示例产生：

2020-07-31 17:05:19.275  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:15
2020-07-31 17:05:19.792  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:15
2020-07-31 17:05:19.793  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox ACK: 15
2020-07-31 17:05:19.805  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:16
2020-07-31 17:05:19.805  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox ACK: 16
2020-07-31 17:05:19.810  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:17
2020-07-31 17:05:20.313  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:17
2020-07-31 17:05:20.313  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox ACK: 17
2020-07-31 17:05:20.318  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:18
2020-07-31 17:05:20.318  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox ACK: 18
2020-07-31 17:05:20.322  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:19
2020-07-31 17:05:20.827  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox Received record topic:event_log_selectingMsgFromMailbox partition:1 offset:19
2020-07-31 17:05:20.828  INFO       17560 --- [ntainer#0-0-C-1] .d.g.a.a.k.c.InboxRetrievalEventConsumer : consumeSelectingMsgFromMailbox ACK: 19

注意：KafkaEventPojo是我的POJO的实现，它按照我们的内部结构保存存储在Kafka中的记录数据-因此您可以根据需要进行更改。此外，上面的代码演示了nack用于单记录侦听器的用法。如果需要批处理选项，可以在提供的文档中找到有关如何进行批处理的示例。

从卡夫卡多次读同一条消息

3 个答案: