Question

我是hadoop的新手。

我正在设置Giraph在带有纱线的hadoop-2.6.5上运行。

当我提交Giraph作业时，作业成功提交但失败了，我在登录容器系统日志下面：

2018-01-30 12：09：01,190 INFO [主要] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：创建MRAppMaster 申请appattempt_1517293264136_0002_000002 2018-01-30 12：09：01,437 WARN [main] org.apache.hadoop.util.NativeCodeLoader：无法为您的平台加载native-hadoop库...使用 builtin-java classes适用的地方2018-01-30 12：09：01,471 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：执行时代币：2018-01-30 12：09：01,471 INFO [主要] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：种类： YARN_AM_RM_TOKEN，服务：，Ident：（appAttemptId {application_id { id：2 cluster_timestamp：1517293264136} attemptId：2} keyId： -1485907628）2018-01-30 12：09：01,583 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：使用mapred newApiCommitter。 2018-01-30 12：09：02,154 INFO [主要] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：OutputCommitter set in config null 2018-01-30 12：09：02,207致命[主要] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：启动时出错 MRAppMaster java.lang.NoClassDefFoundError： io / netty / buffer / ByteBufAllocator at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter（BspOutputFormat.java:62）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster $ 1.call（MRAppMaster.java:470）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster $ 1.call（MRAppMaster.java:452）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader（MRAppMaster.java:1541）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter（MRAppMaster.java:452）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit（MRAppMaster.java:371）在 org.apache.hadoop.service.AbstractService.init（AbstractService.java:163）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster $ 4.run（MRAppMaster.java:1499）在java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject.java:422）at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1692）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster（MRAppMaster.java:1496）在 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main（MRAppMaster.java:1429）引起：java.lang.ClassNotFoundException： io.netty.buffer.ByteBufAllocator at java.net.URLClassLoader.findClass（URLClassLoader.java:381）at java.lang.ClassLoader.loadClass（ClassLoader.java:424）at sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java:331）at at java.lang.ClassLoader.loadClass（ClassLoader.java:357）...还有13个 2018-01-30 12：09：02,209 INFO [main] org.apache.hadoop.util.ExitUtil：退出状态1

日志中的诊断显示以下日志：

由于AM，应用程序application_1517293264136_0002失败了2次 appattempt_1517293264136_0002_000002的容器已退出 exitCode：1要获得更详细的输出，请检查应用程序跟踪页：http://172.16.0.218:8088/proxy/application_1517293264136_0002/Then，单击每个尝试的日志链接。诊断：来自的例外集装箱推出。容器ID：container_1517293264136_0002_02_000001 退出代码：1堆栈跟踪：ExitCodeException exitCode = 1：at org.apache.hadoop.util.Shell.runCommand（Shell.java:575）at org.apache.hadoop.util.Shell.run（Shell.java:478）at org.apache.hadoop.util.Shell $ ShellCommandExecutor.execute（Shell.java:766）在 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer（DefaultContainerExecutor.java:212）在 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call（ContainerLaunch.java:302）在 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call（ContainerLaunch.java:82）在java.util.concurrent.FutureTask.run（FutureTask.java:266）at java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）在 java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:617）在java.lang.Thread.run（Thread.java:745）容器退出时带有非零退出代码1未通过此尝试。申请失败。

它失败的类是io/netty/buffer/ByteBufAllocator，它在netty-all jar中：https://mvnrepository.com/artifact/io.netty/netty-all

从其他问题我尝试在HADOOP_CLASSPATH中添加jar。

Yogin-Patel:hadoop yoginpatel$ echo $HADOOP_CLASSPATH
/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar
Yogin-Patel:hadoop yoginpatel$

它也出现在hadoop classpath中。

Yogin-Patel:hadoop yoginpatel$ hadoop classpath
/Users/yoginpatel/hadoop/etc/hadoop:/Users/yoginpatel/hadoop/share/hadoop/common/lib/*:/Users/yoginpatel/hadoop/share/hadoop/common/*:/Users/yoginpatel/hadoop/share/hadoop/hdfs:/Users/yoginpatel/hadoop/share/hadoop/hdfs/lib/*:/Users/yoginpatel/hadoop/share/hadoop/hdfs/*:/Users/yoginpatel/hadoop/share/hadoop/yarn/lib/*:/Users/yoginpatel/hadoop/share/hadoop/yarn/*:/Users/yoginpatel/hadoop/share/hadoop/mapreduce/lib/*:/Users/yoginpatel/hadoop/share/hadoop/mapreduce/*:/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar:/contrib/capacity-scheduler/*.jar
Yogin-Patel:hadoop yoginpatel$

我正在尝试在开发环境中进行设置。这是单节点设置。

我甚至尝试过

job.addFileToClassPath(new Path("/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar"));

这些方法都没有帮助。如何让hadoop节点获得必要的jar访问？

这是一个GiraphJob提交代码，可以将地图缩减作业提交到群集：

    @Test
    public void testPageRank() throws IOException, ClassNotFoundException, InterruptedException {

        GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
        giraphConf.setWorkerConfiguration(1,1,100);
        GiraphConstants.SPLIT_MASTER_WORKER.set(giraphConf, false);

        giraphConf.setVertexInputFormatClass(JsonLongDoubleFloatDoubleVertexInputFormat.class);
        GiraphFileInputFormat.setVertexInputPath(giraphConf,
                                                 new Path("/input/tiny-graph.txt"));
        giraphConf.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);

        giraphConf.setComputationClass(PageRankComputation.class);

        GiraphJob giraphJob = new GiraphJob(giraphConf, "page-rank");
        giraphJob.getInternalJob().addFileToClassPath(new Path("/Users/yoginpatel/Downloads/gradle-4.3/caches/modules-2/files-2.1/io.netty/netty-all/4.0.43.Final/9781746a179070e886e1fb4b1971a6bbf02061a4/netty-all-4.0.43.Final.jar"));

        FileOutputFormat.setOutputPath(giraphJob.getInternalJob(),
                                       new Path("/output/page-rank2"));
        giraphJob.run(true);
    }

    private Configuration getConf() {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://localhost:9000");

        conf.set("yarn.resourcemanager.address", "localhost:8032");

        // framework is now "yarn", should be defined like this in mapred-site.xm
        conf.set("mapreduce.framework.name", "yarn");
        return conf;
    }

Answer 1

我通过在hadoop lib路径中放置带有依赖项的giraph jar来实现它：

cp giraph-1.3.0-SNAPSHOT-for-hadoop-2.6.5-jar-with-dependencies.jar ~/hadoop/share/hadoop/mapreduce/lib/

FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster：启动MRAppMaster时出错java.lang.NoClassDefFoundError

1 个答案: