减速机后缓慢启动映射器

Question

我正在使用ChainReducer来构建以下模式

Map1 --> Reduce --> Map2 

我希望Map2步骤仅在Reduce步骤完成后启动。 hadoop map中是否有属性reduce来设置它。

Answer 1

您应该做的是以下内容：

ChainMapper.addMapper(generalConf, Map1.class, ..., new JobConf(false));
ChainReducer.setReducer(generalConf, Reduce.class, ..., new JobConf(false));
ChainReducer.addMapper(generalConf, Map2.class, ..., true, new JobConf(false));
// execute the job with generalConf

上述代码将保证Map2的执行只会在Reduce完成时启动。