Question

我正在关注这本书（权威指南hadoop）。

在尝试使用localhost执行本书中的示例时，我遇到了错误，

14/06/13 22:24:57 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****hdfs://localhost/usr/kim/input/ncdc
14/06/13 22:24:57 INFO input.FileInputFormat: Total input paths to process : 3
14/06/13 22:24:57 WARN snappy.LoadSnappy: Snappy native library is available
14/06/13 22:24:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/06/13 22:24:57 INFO snappy.LoadSnappy: Snappy native library loaded
14/06/13 22:24:57 INFO mapred.JobClient: Running job: job_201406132100_0011
14/06/13 22:24:58 INFO mapred.JobClient:  map 0% reduce 0%
14/06/13 22:25:11 INFO mapred.JobClient: Task Id : attempt_201406132100_0011_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mapred.MaxTemperatureMapper_v1
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:867)
    at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mapred.MaxTemperatureMapper_v1
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:865)
    ... 8 more

我使用命令行hadoop jar MaxTemperatureDriver.jar mapred.MaxTemperatureDriver -conf /hadoop_conf/Hadoop-localhost.xml /input/ncdc /max-temp

执行了该作业

jar文件中有2个文件夹META-INF，mapred，mapred文件夹中有5个类（这些类在mapred包中）

MaxTemeratureReducer.class
MaxTemperatureDriver.class
MaxTemperatureMapper_v1.class
MaxTemperatureMapper_v1 $ Temperature.class
NcdcRecordParser.class

那些是配置文件，MaxTemperatureDriver和MaxTemperatureMapper_v1

<?xml version="1.0"?>
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost/</value>
    </property>

    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:8021</value>
    </property>

    <property>
        <name>dfs.replication</name>    
        <value>1</value>
    </property>
</configuration>

public class MaxTemperatureDriver extends Configured implements Tool{
    @Override
    public int run(String[] args) throws Exception{
        if(args.length != 2){
            System.err.printf("Usage: %s [generic options] <input> <output>\n", getClass().getSimpleName());
            ToolRunner.printGenericCommandUsage(System.err);
            return -1;
        }

        Job job = new Job(getConf(), "Max Temperature");
        job.setJarByClass(getClass());

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job,  new Path(args[1]));

        job.setReducerClass(MaxTemeratureReducer.class);
        job.setMapperClass(MaxTemperatureMapper_v1.class);
        job.setCombinerClass(MaxTemeratureReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        return job.waitForCompletion(true) ? 0 : 1;
    }
}

public class MaxTemperatureMapper_v1 extends Mapper<LongWritable, Text, Text, IntWritable>{
    enum Temperature{
        OVER_100
    }

    private NcdcRecordParser parser = new NcdcRecordParser();

    @Override
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
        parser.parse(value);    //Text.toString()
        if(parser.isValidTemperature()){
            int airTemperature = parser.getAirTemperature();
            if(airTemperature > 1000){
                System.err.println("Temperature over 100 degrees for input: " + value);
                context.setStatus("Detected possibly corrupt record: see logs.");
                context.getCounter(Temperature.OVER_100).increment(1);
            }
            context.write(new Text(parser.getYear()), new IntWritable(parser.getAirTemperature()));
        }
    }
}

Answer 1

按类名设置jar，有时会给出错误，并设置inputformat和输出格式，希望这对你有帮助..

job.setJarByClass(MaxTemperatureDriver.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

Answer 2

job.setJarByClass（＆＃34; Main class＆＃34;）;

主类意味着拥有 main（）方法

hadoop，java.lang.RuntimeException：java.lang.ClassNotFoundException错误

2 个答案: