Question

当我从HDInsight群集上的Azure数据工厂V2提交Spark活动时，我收到错误 -

Application application_1524230454528_0060 failed 5 times due to AM Container for appattempt_1524230454528_0060_000005 exited with exitCode: -1000
    For more detailed output, check the application tracking page: http://hn1-rbc-cl.tsgjwtochfbexi5wvtaxuipmmg.rx.internal.cloudapp.net:8088/cluster/app/application_1524230454528_0060 Then click on links to logs of each attempt.
    Diagnostics: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    java.io.FileNotFoundException: wasb://rbc-cluster-2018-04-20t13-11-42-670z@rbcdemo.blob.core.windows.net/user/livy/.sparkStaging/application_1524230454528_0060/__spark_conf__.zip: No such file or directory.
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
    at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
    at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
    at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
    at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Failing this attempt. Failing the application.

有人可以帮我弄清楚问题是什么，或者有任何其他解决方法通过Azure数据工厂管道运行spark作业。我坚持这个，无法继续。根据Azure Data Factory文档，我尝试将我的文件放在容器中的./files文件夹中，其中包含我的spark jar。但仍面临同样的问题。

Answer 1

您的HDI群集版本和用于开发程序的Java版本是什么，请确保版本相同。

BTW，HDI群集3.5+需要java 8，而早期版本需要java 7。

__spark_conf__.zip：没有这样的文件或目录

1 个答案: