Hadoop Streaming作业错误

时间:2017-02-06 10:15:03

标签: python linux hadoop hadoop-streaming amazon-emr

您好我试图在EMR 4.7.2中使用Python运行Hadoop Streaming作业,命令如下:

hadoop-streaming -archives s3://mybucket/scripts/HDP/python_scripts/py.tgz -mapper py.tgz/processRAWData.py -input /data/raw_combined/ -output s3://ccv-eu01-private/temp/logoptemp/ -jobconf mapred.task.timeout=14000000 -jobconf mapred.tasktracker.expiry.interval=14000000 -jobconf mapred.reduce.tasks=0 -jobconf mapred.map.tasks=15000

这项工作用于平稳运行3个月,但现在它失败了&以下是日志:

caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    ... 17 more
Caused by: java.lang.RuntimeException: configuration exception
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:232)
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
    ... 22 more
Caused by: java.io.IOException: Cannot run program "/mnt1/yarn/usercache/hadoop/appcache/application_1486363893937_0003/container_1486363893937_0003_01_000055/./py.tgz/processRAWData_unix.py": error=26, Text file busy
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:219)
    ... 23 more
Caused by: java.io.IOException: error=26, Text file busy
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 24 more

我检查了解决上述错误,但我无法解决错误的根本原因。为什么会出现此错误?任何人都可以帮我解决这个错误的解决方案吗?谢谢

0 个答案:

没有答案
相关问题