Spark Streaming Checkpoint异常

Question

我有我的火花流媒体管道与kafka集成，我也配置了检查指向，为了测试弹性我手动杀死了工作然后重新启动它然后我得到低于异常

Exception in thread "main" org.apache.spark.SparkException: org.apache.spark.streaming.dstream.ShuffledDStream@1d304ac has not been initialized
    at org.apache.spark.streaming.dstream.DStream.isTimeValid(DStream.scala:267)

我使用的代码

JavaStreamingContextFactory contextFactory = new JavaStreamingContextFactory() {


      public JavaStreamingContext create() {
          final SparkConf sparkConf = new SparkConf().setAppName("JavaKafkaWordCount");
            sparkConf.setMaster("local[2]");

        JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));  // new context

        jssc.checkpoint("D:\\Checkpoint");
        return jssc;
      }
    };

    // Get JavaStreamingContext from checkpoint data or create a new one
    JavaStreamingContext jssc = JavaStreamingContext.getOrCreate("D:\\Checkpoint", contextFactory);
    int numThreads = Integer.parseInt(1+"");

请建议我做错了什么

Answer 1

private static JavaStreamingContext createContext(){
    final SparkConf sparkConf = new SparkConf().setAppName("JavaKafkaWordCount");
    sparkConf.setMaster("local[2]");

    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));  // new context

    jssc.checkpoint("D:\\Checkpoint");
    return jssc;
}

Function0<JavaStreamingContext> createContextFunc = new Function0<JavaStreamingContext>() {
    @Override
    public JavaStreamingContext call() {
        return createContext();
    }
};

JavaStreamingContext streamingContext = JavaStreamingContext.getOrCreate("D:\\Checkpoint", createContextFunc);

使用Function0，这对我有用。

Answer 2

http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing

您没有显示创建DStreams /转换的代码。把它们放在create（）方法中。