Question

我试图了解MapReduce如何对Map输出键进行排序，以及它使用的排序算法是什么。我有一个像这样的文本文件

a b e f c b

如何使用这些键执行排序。我实现了一个扩展WritableComparator接口的自定义SortComparator类。我想看看排序是如何发生的，所以我将操作写入文件。

public static class MySortComparator2 extends WritableComparator{

  @Override
  public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {

      Text x=new Text("");
      Text y=new Text("");
      try {
        x.readFields(new DataInputStream(new ByteArrayInputStream(b1)));
        y.readFields(new DataInputStream(new ByteArrayInputStream(b2)));

        FileWriter writerOfComparisions=new FileWriter("/home/srimanth/Comparisons",true);
        writerOfComparisions.write("Comparing "+x.toString()+" with "+y.toString());
        writerOfComparisions.write(" ----> returning "+x.compareTo(y));
        writerOfComparisions.write("\n");
        writerOfComparisions.close();

    } catch (IOException e) {
        e.printStackTrace();
    }
      return x.compareTo(y);
  }

Comparisons文件的输出是这个

Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing a with a ----> returning 0
Comparing b with a ----> returning 1
Comparing c with a ----> returning 2
Comparing f with a ----> returning 5
Comparing e with a ----> returning 4
Comparing b with a ----> returning 1
Comparing b with c ----> returning -1
Comparing c with f ----> returning -3
Comparing f with e ----> returning 1
Comparing e with b ----> returning 3
Comparing b with a ----> returning 1

这个排序算法是什么，我的wordcount的最终输出是这个

看起来mapreduce没有正确发生。这是我的Mapper和Reducer类。

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());

      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);

      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();    
    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {


      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }

      result.set(sum);
      context.write(key, result);
    }
  }

我做错了什么。我将不胜感激。在此先感谢。

Answer 1

您是否在驱动程序代码中设置了以下内容？

job.setSortComparatorClass（MySortComparator2.class）;

自定义SortComparator无法在MapReduce wordcount程序

1 个答案: