Kafka消费者/生产者API

时间:2017-07-25 05:34:34

标签: apache-kafka kafka-consumer-api apache-kafka-streams

如何计算已向Kafka中的主题发送了多少总消息以及当时消费者已消费或承诺了多少消息?

我正在启动kafka连接器 -

Map<String, String> kafkaParams = new HashMap<>();
kafkaParams.put("metadata.broker.list", "localhost:9092");
Set<String> topics = Collections.singleton("mytopic");

JavaPairInputDStream<String, String> directKafkaStream = 
KafkaUtils.createDirectStream(ssc,
   String.class, String.class, StringDecoder.class, StringDecoder.class, 
   kafkaParams, topics);

处理为 -

directKafkaStream.foreachRDD(rdd -> {
 System.out.println("--- New RDD with " + rdd.partitions().size()
        + " partitions and " + rdd.count() + " records");
 rdd.foreach(record -> System.out.println(record._2));
});

持续2秒

--- New RDD with 2 partitions and 3 records
value-1
value-0
value-2
--- New RDD with 2 partitions and 7 records
value-3
value-5
value-7
value-9
value-4
value-6
value-8
--- New RDD with 2 partitions and 8 records
value-11
value-10
value-13
...

0 个答案:

没有答案