Pig UDF从UDF抛出错误捕获错误:GetCounty,超出范围访问[1]

时间:2014-12-29 04:47:15

标签: apache-pig

我正在写一个猪程序,它读取包含city,zip的文件,然后将城市传递给UDF。 UDF将在哈希映射中加载包含县,城市的文件。然后UDF从哈希映射中找到城市的县并返回它。

请让我知道我在这里做错了什么;运行程序时出现以下错误:

2014-12-28 16:15:16,506 WARN org.apache.hadoop.mapred.Child: Error running child
org.apache.pig.backend.executionengine.ExecException: ERROR 2078: Caught error from UDF: GetCounty, Out of bounds access [1]
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:370)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:434)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:340)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at GetCounty.exec(GetCounty.java:33)
at GetCounty.exec(GetCounty.java:1)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
... 15 more
2014-12-28 16:15:16,510 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

输入文件包含以下数据:

File zipcity:
irving  75038
san francisco   94903
san rafael      94905
las vegas       98043
coppel  75063

File citycounty:
irving  dallas
las vegas       tarrant
san francisco   san francisco
coppel  dallas

public class GetCounty extends EvalFunc<String> {
String lookupfile;
HashMap<String, String> lookup = null;

public String exec(Tuple input) throws IOException{
    if ( input.size() != 1 ){       
        return null;
    }

    if ( lookup == null ) {
        FileSystem fs = FileSystem.get(UDFContext.getUDFContext().getJobConf());
        DataInputStream in = fs.open(new Path(lookupfile));
        String line;
        while ( (line = in.readLine()) != null){
            String[] tok = new String[2];
            tok = line.split(":", 2);
            lookup.put(tok[0], tok[1]);
        }}

    String city = (String) input.get(0);        
    return lookup.get(city);        
}

public GetCounty(String f){
    lookupfile = f;
}
}

我按照以下方式调用猪:

grunt> register 'PigMyUDF.jar';
grunt> define GetCounty GetCounty('pig/citycounty');
grunt> a = load 'pig/zipcity' as ( city:chararray, zip:int );
grunt> b = foreach a generate city, zip, GetCounty(city);
grunt> dump b;

1 个答案:

答案 0 :(得分:0)

你能试试吗?输入字段由制表符分隔。

<强> zipcity

irving  75038
san francisco   94903
san rafael      94905
las vegas       98043
coppel  75063

<强> citycounty

irving  dallas
las vegas       tarrant
san francisco   san francisco
coppel  dallas

<强> PigScript:

A = LOAD 'zipcity' AS (city:chararray, zip:int);
B = LOAD 'citycounty' AS  (city:chararray,country:chararray);
C = JOIN A BY city,B BY city;
D = FOREACH C GENERATE A::city AS city,A::zip AS zip,B::country AS country;
DUMP D;

<强>输出:

(coppel,75063,dallas)
(irving,75038,dallas)
(las vegas,98043,tarrant)
(san francisco,94903,san francisco)