Question

我正在运行多个单独的kafka流。为此，我创建了一个流管理器来保存这些流。本质上，下面是SM类。

public class StreamManager {

public Map<String, BaseStream> streamMap = new HashMap<String, BaseStream>();
public Map<String, ReadOnlyKeyValueStore<Long, BaseModel>> storeMap = new HashMap<String, ReadOnlyKeyValueStore<Long, BaseModel>>();    

public StreamManager(String bootstrapServer){
    initialize(bootstrapServer);
}

/**
 * initialize streams.  Right now hard coding creation of streams here.
 * @param bootstrapServer
 */
public void initialize(String bootstrapServer){

    Properties s1Props = this.GetStreamingProperties(bootstrapServer, "STREAM_1");
    Properties s2Props = this.GetStreamingProperties(bootstrapServer, "STREAM_1");

    BaseStream s1Stream = new CompositeInfoStream(new KStreamBuilder(), compositeInfoProps);
    BaseStream s2Stream = new ImcInfoStream(new KStreamBuilder(), imcInfoProps);

    streamMap.put(s1Stream.storeName, s1Stream);
    streamMap.put(s2Stream.storeName, s2Stream);

/**
 * Start all streams
 */
public void startStreams(){
    for(BaseStream stream: streamMap.values()){
        stream.start();
    }
}

public static void main(String[] args) throws Exception {

    StreamManager mgr = new StreamManager(StreamingConfig.instance().MESSAGE_SERVER);
    StreamManager.startStreamServices(mgr);

    Runtime.getRuntime().addShutdownHook(new Thread() {
        @Override
        public void run() {
            try {

            } finally {
                mgr.closeStreams();

            }
        }
    });

    int i = 0;
    while(true) {
        if (i++ == 0)
            mgr.logAllStreamStates();

        Thread.sleep(60000);
        if (i == 60) i = 0;
    }       
}
}

我初始化并启动流，然后让进程循环运行。现在，我想要对单个流进行更多控制。要启动并杀死它们（如果需要的话）（出于某种奇怪的原因，我的流经常进入REBALANCE模式并且不返回。目前，如果其中一个流进入REBALANCING，则必须杀死整个SM（所有流）并重新启动我只想重启单个流。

我想了解我的体系结构应该如何。 kafka流是否提供管理流集群的机制？我可以使用多处理来完成此任务吗？如果可以，请记住我使用Windows进行开发，使用Linux进行部署

Kafka Streams：用于处理多个流的程序体系结构

0 个答案: