Question

我正在尝试运行一个程序，按照此链接中给出的步骤计算其频率的字数：http://developer.yahoo.com/hadoop/tutorial/module3.html

我已经加载了一个名为 input 的目录，其中包含三个文本文件。

我能够正确配置所有内容。现在，在运行WordCount.java时，我在输出目录中的 part-00000 文件中看不到任何内容。

Mapper的java代码是：

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordCountMapper extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {

private final IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(WritableComparable key, Writable value,
  OutputCollector output, Reporter reporter) throws IOException {

String line = value.toString();
StringTokenizer itr = new StringTokenizer(line.toLowerCase());
while(itr.hasMoreTokens()) {
  word.set(itr.nextToken());
  output.collect(word, one);
}
}

@Override
public void map(LongWritable arg0, Text arg1,
    OutputCollector<Text, IntWritable> arg2, Reporter arg3)
     throws IOException {
// TODO Auto-generated method stub

 }

}

减少代码是：

public class WordCountReducer extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {

public void reduce(Text key, Iterator values,
  OutputCollector output, Reporter reporter) throws IOException {

int sum = 0;
while (values.hasNext()) {
    //System.out.println(values.next());
  IntWritable value = (IntWritable) values.next();
  sum += value.get(); // process value
}

output.collect(key, new IntWritable(sum));
 }
 }

Word计数器的代码是：

public class Counter {

public static void main(String[] args) {
    JobClient client = new JobClient();
    JobConf conf = new JobConf(com.example.Counter.class);

    // TODO: specify output types
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    // TODO: specify input and output DIRECTORIES (not files)
    conf.setInputPath(new Path("src"));
    conf.setOutputPath(new Path("out"));

    // TODO: specify a mapper
    conf.setMapperClass(org.apache.hadoop.mapred.lib.IdentityMapper.class);

    // TODO: specify a reducer
    conf
                   .setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);

    client.setConf(conf);
    try {
        JobClient.runJob(conf);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

}

在控制台中我得到这些日志：

13/09/10 10:09:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing the       arguments. Applications should implement Tool for the same.
13/09/10 10:09:20 INFO mapred.FileInputFormat: Total input paths to process : 3
13/09/10 10:09:20 INFO mapred.FileInputFormat: Total input paths to process : 3
13/09/10 10:09:20 INFO mapred.JobClient: Running job: job_201309100855_0012
13/09/10 10:09:21 INFO mapred.JobClient:  map 0% reduce 0%
13/09/10 10:09:25 INFO mapred.JobClient:  map 25% reduce 0%
13/09/10 10:09:26 INFO mapred.JobClient:  map 75% reduce 0%
13/09/10 10:09:27 INFO mapred.JobClient:  map 100% reduce 0%
13/09/10 10:09:35 INFO mapred.JobClient: Job complete: job_201309100855_0012
13/09/10 10:09:35 INFO mapred.JobClient: Counters: 15
13/09/10 10:09:35 INFO mapred.JobClient:   File Systems
13/09/10 10:09:35 INFO mapred.JobClient:     HDFS bytes read=54049
13/09/10 10:09:35 INFO mapred.JobClient:     Local bytes read=14
13/09/10 10:09:35 INFO mapred.JobClient:     Local bytes written=214
13/09/10 10:09:35 INFO mapred.JobClient:   Job Counters 
13/09/10 10:09:35 INFO mapred.JobClient:     Launched reduce tasks=1
13/09/10 10:09:35 INFO mapred.JobClient:     Launched map tasks=4
13/09/10 10:09:35 INFO mapred.JobClient:     Data-local map tasks=4
13/09/10 10:09:35 INFO mapred.JobClient:   Map-Reduce Framework
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce input groups=0
13/09/10 10:09:35 INFO mapred.JobClient:     Combine output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map input records=326
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map output bytes=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map input bytes=50752
13/09/10 10:09:35 INFO mapred.JobClient:     Combine input records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Map output records=0
13/09/10 10:09:35 INFO mapred.JobClient:     Reduce input records=0

我在Hadoop中很新。

请回答适当的答案。

感谢。

Answer 1

Mapper类中有两个map方法。具有@Override注释的那个是实际被覆盖的方法，并且该方法不执行任何操作。因此，映射器中没有任何内容，也没有任何内容进入reducer，因此没有输出。

删除标有map注释的@Override方法，并使用map标记第一个@Override方法。然后修复任何方法签名问题，它应该可以工作。

Answer 2

我遇到了同样的问题。我通过删除覆盖的map方法并将map方法的签名更改为第一个参数为LongWritable来解决它。更新地图方法签名，如下所示：

@Override
public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) 
    throws IOException {

Hadoop - WordCount的结果不是写在输出文件上

2 个答案: