在DSE 4.8 Spark群集上运行Python作业时出错。

时间:2016-04-04 13:01:49

标签: apache-spark pyspark datastax-enterprise

我刚刚提出了在DSE集群4.8上运行Spark(pyspark)的奇怪问题。出于某种原因,当作业运行时,我有几十个错误,如下所述。有人知道有什么办法可以关闭这个错误吗?

我正在使用dse spark-submit命令。

ERROR 2016-04-04 12:49:21,305 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 747
ERROR 2016-04-04 12:49:21,305 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 748
ERROR 2016-04-04 12:49:21,347 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 749
ERROR 2016-04-04 12:49:21,388 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 750

1 个答案:

答案 0 :(得分:0)

When I see this message "Asked to remove non-existent executor" that means the executor crashed for a different reason and so obviously spark can't kill it. Look at the executor logs to find out why it has crashed.

It's also possible you have a networking problem and your driver and executor can't communicate. Have also seen this message in that case.

相关问题