YARN：无法找到或加载主类org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Question

我尝试在Fedora上执行任何M / R2作业时遇到此异常。 Hadoop 2.7.3和2.8.0也有同样的问题。这包括Hive。

[hadoop@master hadoop]$ yarn classpath
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf:/opt/hadoop/hadoop/share/hadoop/common/lib/*
/opt/hadoop/hadoop/share/hadoop/common/*
/opt/hadoop/hadoop/share/hadoop/hdfs
/opt/hadoop/hadoop/share/hadoop/hdfs/lib/*
/opt/hadoop/hadoop/share/hadoop/hdfs/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*
/opt/hadoop/hadoop/contrib/capacity-scheduler/*.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*

mapred-site.xml中

<configuration>

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<property> 
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

和yarn-site.xml

<configuration>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

最后但并非最不重要的是，env setup：

export HADOOP_HOME=/opt/hadoop/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

我很确定我错过了一些明显的东西。我已多次设置它，但必须关闭一些东西。

缺少的类实际上是类路径中jar的一部分。 / opt / hadoop / hadoop文件夹由用户hadoop拥有，并具有所需的所有访问权限。

Answer 1

设置YARN_HOME=$HADOOP_HOME而不是HADOOP_YARN_HOME=$HADOOP_HOME

Answer 2

我遇到了同样的问题（JAVA -1.8 291u ，Hadoop -2.8.0），在 yarn-site.xml 中设置属性 - YARN 应用程序类路径后解决了。

第一步：执行“hadoop classpath”。

此命令显示要作为yarn-site.xml中的值传递的路径列表

第 2 步：编辑 yarn-site.xml 如下：

 <property>
    <name>yarn.application.classpath</name>
    <value>output from step1 </value>
 </property>

在触发 mapreduce 作业之前重新启动 yarn 脚本。