执行PIG脚本时出错

时间：2015-10-22 07:17:48

标签： hadoop mapreduce apache-pig

我正在谷歌云Hadoop环境中运行一个猪脚本     pig -useHCatalog -x mapreduce -f profile.pig 我有两个表，每个表有50,000个记录将被交叉并与一个10,00,000的表连接。我运行相同的脚本少记录它运行得很好但是当我增加记录数时它会抛出这个错误。

    2015-10-22 05:38:56,261 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,266 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history serv
er
2015-10-22 05:38:56,377 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 0: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exceptio
n from backed error: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
Killed by external signal
2015-10-22 05:38:56,377 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2015-10-22 05:38:56,380 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics: 
HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
2.6.0.2.2.8.0-3150      0.14.0.2.2.8.0-3150     hdfs    2015-10-22 05:34:17     2015-10-22 05:38:56     HASH_JOIN,GROUP_BY,FILTER,CROSS,UNION
Some jobs have failed! Stop running all dependent jobs
Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTime      AvgMapTime      MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime        Alias   Feature Outp
uts
job_1444401341866_0534  1       0       11      11      11      11      0       0       0       0       t_female,t_male,t_profile       MULTI_QUERY,MAP_ONLY
job_1444401341866_0535  2       1       6       6       6       6       91      91      91      91      t_female,t_male,t_raid_female,t_raid_female1
job_1444401341866_0536  2       1       25      25      25      25      89      89      89      89      t_female,t_male,t_raid_male,t_raid_male1
job_1444401341866_0537  2       0       5       5       5       5       0       0       0       0       t_female,t_male,t_mf_union      MAP_ONLY
Failed Jobs:
JobId   Alias   Feature Message Outputs
job_1444401341866_0538  j_ci1,j_mf_education,j_mf_height,j_mf_occupation,j_mf_religion,j_mf_weight,t_ci1,t_mf_transpose,t_mf_union,t_raid       HASH_JOIN       Message: Job failed!
Input(s):
Successfully read 10001 records (1509451 bytes) from: "matrimony.profile_gce_limit"Input(s):
Successfully read 10001 records (1509451 bytes) from: "matrimony.profile_gce_limit"

Output(s):

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1444401341866_0534  ->      job_1444401341866_0535,job_1444401341866_0536,job_1444401341866_0537,
job_1444401341866_0535  ->      job_1444401341866_0538,
job_1444401341866_0536  ->      job_1444401341866_0538,
job_1444401341866_0537  ->      job_1444401341866_0538,
job_1444401341866_0538  ->      null,
null    ->      null,
null    ->      null,
null


2015-10-22 05:38:56,455 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/timeline/
2015-10-22 05:38:56,456 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,459 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-10-22 05:38:56,572 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/timeline/
2015-10-22 05:38:56,572 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,576 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-10-22 05:38:56,675 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/timeline/
2015-10-22 05:38:56,676 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,679 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-10-22 05:38:56,780 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/timeline/
2015-10-22 05:38:56,780 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,783 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-10-22 05:38:56,883 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/timeline/
2015-10-22 05:38:56,883 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,886 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2015-10-22 05:38:56,981 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/Backend error message
timeline/
2015-10-22 05:38:56,982 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:56,985 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,083 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,083 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,086 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,182 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,182 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,185 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,275 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,275 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,278 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,370 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,370 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,373 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,475 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,475 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,478 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,570 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://hadoop-w-0.c.horton-cluster-3.internal:8188/ws/v1/
timeline/
2015-10-22 05:38:57,570 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop-w-0.c.horton-cluster-3.internal/10.240.0.3:8050
2015-10-22 05:38:57,574 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history s
erver
2015-10-22 05:38:57,601 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Some jobs have failed! Stop running all dependent jobs
2015-10-22 05:38:57,602 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2997: Unable to recreate exception from backed error: Container killed on request. Exit code is 
137
Container exited with a non-zero exit code 137
Killed by external signal
Details at logfile: /home/hdfs/workfile/pig_1445492051329.log
2015-10-22 05:38:57,603 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/hdfs/workfile/pig_1445492051329.log
2015-10-22 05:38:57,603 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/hdfs/workfile/pig_1445492051329.log
2015-10-22 05:38:57,603 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/hdfs/workfile/pig_1445492051329.log
2015-10-22 05:38:57,623 [main] INFO  org.apache.pig.Main - Pig script completed in 4 minutes, 46 seconds and 401 milliseconds (286401 ms)


And this is what it is in the log file

    ================================================================================
Pig Stack Trace
---------------
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:179)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
================================================================================
Pig Stack Trace
---------------
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:179)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:170)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
================================================================================

1 个答案:

答案 0 :(得分：0)

就我而言，是struct Foo { Foo(...) {...} #if _MSC_FULL_VER == 150030729 // Visual Studio 2008, SP1 Foo() {} //<- work-around for VS2008 SP1 bug #endif }; 。

我到达了OrcStorage('\t')，并且有效。

相关问题
最新问题