我有一个kafka主题,我正在通过Kafka Producer发送数据。现在在消费者方面,我有两种选择。
1。使用KafkaConsumer-kafkaConsumer的代码如下,该代码从主题读取数据并正常工作。
@EnableKafka
@Configuration
@PropertySource("kaafka.properties")
public class RawEventKafkaConsumer {
private static final Logger logger = LoggerFactory.getLogger(RawEventKafkaConsumer.class);
@Autowired
private DataModelServiceImpl dataModelServiceImpl;
private PolicyExecutor policyExecutor;
public RawEventKafkaConsumer() {
policyExecutor = new PolicyExecutor();
}
@Value("${spring.kafka.topic}")
private String rawEventTopicName;
@Value("${spring.kafka.consumer.auto-offset-reset}")
private String autoOffsetReset;
@Value("${spring.kafka.consumer.bootstrap-servers}")
private String bootStrapServer;
@Value("${spring.kafka.consumer.group-id}")
private String groupId;
@Value("${spring.kafka.producer.key-serializer}")
private String keySerializer;
@Value("${spring.kafka.producer.value-serializer}")
private String valueSerializer;
@Value("${spring.kafka.consumer.key-deserializer}")
private String keyDeserializer;
@Value("${spring.kafka.consumer.value-deserializer}")
private String valueDeserializer;
@Bean
public DefaultKafkaConsumerFactory<String, BaseDataModel> rawEventConsumer() {
Map<String, Object> consumerProperties = new HashMap<>();
consumerProperties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootStrapServer);
consumerProperties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer);
consumerProperties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, valueDeserializer);
consumerProperties.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
consumerProperties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetReset);
consumerProperties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
return new DefaultKafkaConsumerFactory<>(consumerProperties);
}
@Bean(name="kafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<String, BaseDataModel> kafkaListenerContainerFactory() {
logger.info("kafkaListenerContainerFactory called..");
ConcurrentKafkaListenerContainerFactory<String, BaseDataModel> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(rawEventConsumer());
return factory;
}
@KafkaListener(topics = "rawEventTopic", containerFactory = "kafkaListenerContainerFactory")
public void listen(String baseDataModel) {
ObjectMapper mapper = new ObjectMapper();
BaseDataModel csvDataModel;
try {
csvDataModel = mapper.readValue(baseDataModel, BaseDataModel.class);
//saving the datamodel in elastic search.
//dataModelServiceImpl.save(csvDataModel);
System.out.println("Message received " + csvDataModel.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
2。使用Spark Stream消耗kafkaTopic数据-代码如下-
@Service
public class RawEventSparkStreamConsumer {
private final Logger logger = LoggerFactory.getLogger(RawEventSparkStreamConsumer.class);
@Autowired
private DataModelServiceImpl dataModelServiceImpl;
@Autowired
private JavaStreamingContext streamingContext;
@Autowired
private JavaInputDStream<ConsumerRecord<String, String>> messages;
@PostConstruct
private void sparkRawEventConsumer() {
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.execute(()->{
messages.foreachRDD((rdd) -> {
System.out.println("RDD coming *************************______________________________---------------------.." + rdd.count());
rdd.foreach(record -> {
System.out.println("Data is comming...." + record);
});
});
streamingContext.start();
try {
streamingContext.awaitTermination();
} catch (InterruptedException e) { // TODO Auto-generated catch block
e.printStackTrace();
}
});
}
}
使用者kafka使用者和Spark流均成功读取了主题中的数据。现在我有一个问题,如果两者都在做同一件事(从主题中读取数据),那么
谢谢。
答案 0 :(得分:2)
简而言之,与Kafka Consumer仅在单个JVM中运行相比,您需要一个Spark集群以分布式方式运行Spark代码,并且您需要手动运行同一应用程序的多个实例以进行扩展。
换句话说,您将以不同的方式运行它们。 spark-submit
与java -jar
。我不相信使用Spring更改
另一个区别是,“普通用户”对Kafka配置有更多控制权,并且一次获得一条记录。 Spark RDD可以有很多事件,并且它们都必须具有相同的“模式”,除非您需要复杂的解析逻辑,使用RDD对象编写这些逻辑要比使用为您提取的ConsumerRecord
值更难。
总的来说,将它们结合起来并不是一个好主意。
如果他们从同一个主题中读取内容,那么Kafka Consumer协议只能为每个分区分配一个消费者...尚不清楚您的主题有多少个分区,但这可以解释为什么一个分区可以工作,但不能另一个