跑步中的火花，但纱线无法找到它

时间：2016-10-12 08:02:37

标签： hadoop apache-spark yarn

我有关于纱线簇的问题 我在localhost运行hdfs-namenode，hdfs-datanode，yarn，然后在localhost上运行spark-master和spark-worker，如下所示：

$ jps  
5809 Main  
53730 ResourceManager  
53540 SecondaryNameNode  
53125 NameNode  
56710 Master  
54009 NodeManager  
56809 Worker  
53308 DataNode  
56911 Jps

我可以看到spark-worker链接到spark-master throw http://127.0.0.1:8080
img：spark-web-ui

[！[在此处输入图像说明] [1]] [1]

但是在纱线web-ui http://127.0.0.1:8088中，Nodes of the cluster页面中没有任何内容 img：

[！[在此处输入图像说明] [2]] [2]

我的conf/spark-env.sh是

export SCALA_HOME="/opt/scala-2.11.8/"  
export JAVA_HOME="/opt/jdk1.8.0_101/"  
export HADOOP_HOME="/opt/hadoop-2.7.3/"  
export HADOOP_CONF_DIR="/opt/hadoop-2.7.3/etc/hadoop/"  
export SPARK_MASTER_IP=127.0.0.1  
export SPARK_LOCAL_DIRS="/opt/spark-2.0.0-bin-hadoop2.7/"  
export SPARK_DRIVER_MEMORY=1G

conf/spark-defaults.conf是

spark.master                            spark://127.0.0.1:7077  
spark.yarn.submit.waitAppCompletion     false  
spark.yarn.access.namenodes             hdfs://127.0.0.1:8032

yarn-site.xml是

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>127.0.0.1</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8035</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
</configuration>

当我提交申请时使用
spark-submit --master yarn --deploy-mode cluster test.py
我可以像这样出来

16/10/12 16:19:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/12 16:19:30 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
16/10/12 16:19:30 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
16/10/12 16:19:30 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
16/10/12 16:19:30 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/10/12 16:19:30 INFO yarn.Client: Setting up container launch context for our AM
16/10/12 16:19:30 INFO yarn.Client: Setting up the launch environment for our AM container
16/10/12 16:19:30 INFO yarn.Client: Preparing resources for our AM container
16/10/12 16:19:31 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
16/10/12 16:19:32 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909/__spark_libs__2140674596658903486.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/__spark_libs__2140674596658903486.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/home/fuxiuyin/PycharmProjects/spark-test/test.py -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/test.py
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/pyspark.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/py4j-0.10.1-src.zip
16/10/12 16:19:33 INFO yarn.Client: Uploading resource file:/opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909/__spark_conf__3570291475444079549.zip -> hdfs://127.0.0.1:9000/user/fuxiuyin/.sparkStaging/application_1476256306830_0002/__spark_conf__.zip
16/10/12 16:19:33 INFO spark.SecurityManager: Changing view acls to: fuxiuyin
16/10/12 16:19:33 INFO spark.SecurityManager: Changing modify acls to: fuxiuyin
16/10/12 16:19:33 INFO spark.SecurityManager: Changing view acls groups to:
16/10/12 16:19:33 INFO spark.SecurityManager: Changing modify acls groups to:
16/10/12 16:19:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(fuxiuyin); groups with view permissions: Set(); users  with modify permissions: Set(fuxiuyin); groups with modify permissions: Set()
16/10/12 16:19:33 INFO yarn.Client: Submitting application application_1476256306830_0002 to ResourceManager
16/10/12 16:19:33 INFO impl.YarnClientImpl: Submitted application application_1476256306830_0002
16/10/12 16:19:33 INFO yarn.Client: Application report for application_1476256306830_0002 (state: ACCEPTED)
16/10/12 16:19:33 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1476260373944
     final status: UNDEFINED
     tracking URL: http://localhost:8088/proxy/application_1476256306830_0002/
     user: fuxiuyin
16/10/12 16:19:33 INFO util.ShutdownHookManager: Shutdown hook called
16/10/12 16:19:33 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.0.0-bin-hadoop2.7/spark-3cdb2435-d6a0-4ce0-a54a-f2849d5f4909

这是成功的，但在使用纱线web-ui时，这个应用程序并不总是在ACCEPTED中运行 enter image description here

看起来没有火花节点运行此应用程序。

谁能告诉我什么是错的？ 感谢〜

2 个答案:

答案 0 :(得分：0)

您可以指定一种类型的群集：

  • YARN（群集或客户端模式）
  • Spark standalone
  • Mesos

您已启动Spark独立服务器，并且您正在连接到此群集管理器。如果你想在YARN上启动Spark，你应该指定yarn master - 只需--master yarn

编辑： 请添加日志和spark-submit命令。还请发布你是如何推出YARN的。如果第一次尝试是错误的，那么这意味着您有配置问题

编辑编号2：似乎YARN没有足够的资源来处理您的应用程序。请检查您的配置，即检查增加yarn.nodemanager.resource.memory-mb是否有帮助。您也可以访问Spark Web UI - http://application-master-ip:4040 - 并查看Spark Context中的信息。

此外，您可以检查是否可以将应用程序部署到Spark Standalone（您也开始使用），只需在配置中设置--master spark://...:即可。然后你会确定它是YARN还是Spark中的问题

顺便说一句。如果你提交给YARN，你可以省略运行Spark Standalone :)并且Stanalone Workers使用的内存可以被YARN使用

答案 1 :(得分：0)

:)。谢谢大家，我很抱歉浪费你的时间。当我检查http://localhost:8088/中的资源时，我注意到了这一点：

Unhealthy Nodes

我只是停止服务器并删除tmp目录和日志目录。然后它工作。

再次感谢您

相关问题
最新问题