在Docker中启动工作节点并连接到在主机OS上运行的主节点

Question

我正在尝试以独立模式运行spark。主节点和工作节点已启动并在主机操作系统上运行。

我正在尝试启动一个docker容器以作为工作程序节点运行。主机操作系统是ubuntu 18.04 64位。 容器Dockerfile如下所示，它将在Alpine Linux上运行。

### Dockerfile for creating images of spark worker


#set the base image as alpine-java 
# headless openjdk8.
FROM anapsix/alpine-java

#install few required dependencies in the alpine linux os
#To upgrade all the packages of a running system, use upgrade
#install wget to download the hadoop,spark binaries
#install git  as all the required softwares for  alpine are in git repos
#install unzip to unzip the downloaded  files
#Py4J enables Python programs running in a Python interpreter
#to dynamically access java objects in a JVM.
RUN apk update --no-cache && apk upgrade --no-cache && \
    apk add --no-cache wget \
            git \
            unzip \
            python3 \
            python3-dev && \
            pip3 install --no-cache-dir --upgrade pip -U py4j && \
            cd /home && \
            wget http://www-eu.apache.org/dist/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz && \
            tar -xvf spark-2.3.1-bin-hadoop2.7.tgz && \
            rm -rf spark-2.3.1-bin-hadoop2.7.tgz && \
            rm -rf /var/cache/* && \
            rm -rf /root/.cache/*

# set some enviroment variables for the alpine

# setting the seed value of hash randomization to an integer
ENV PYTHONHASHSEED 2

ENV SPARK_HOME /home/spark-2.3.1-bin-hadoop2.7
ENV PYSPARK_PYTHON python3
ENV PATH $PATH:$SPARK_HOME/bin
WORKDIR $SPARK_HOME
ENTRYPOINT $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker $MYMASTER

使用以下命令使用上述Dockerfile创建映像

docker build -t spkworker .

图像创建成功

问题是在使用以下命令启动工作节点时 Dockerfile具有变量$MYMASTER，该变量应该传递主URL来部署工作程序。

运行命令如下所示，我正在环境变量中传递主节点URL。

docker run spkworker --name worker1 --env MYMASTER=spark://127.0.1.1:7077

它失败，并显示错误消息

2018-08-05 18:00:57 INFO  Worker:2611 - Started daemon with process name: 8@44bb0d682a48
2018-08-05 18:00:57 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-08-05 18:00:57 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-08-05 18:00:57 INFO  SignalUtils:54 - Registered signal handler for INT
Usage: Worker [options] <master>

Master must be a URL of the form spark://hostname:port

Options:
  -c CORES, --cores CORES  Number of cores to use
  -m MEM, --memory MEM     Amount of memory to use (e.g. 1000M, 2G)
  -d DIR, --work-dir DIR   Directory to run apps in (default: SPARK_HOME/work)
  -i HOST, --ip IP         Hostname to listen on (deprecated, please use --host or -h)
  -h HOST, --host HOST     Hostname to listen on
  -p PORT, --port PORT     Port to listen on (default: random)
  --webui-port PORT        Port for web UI (default: 8081)
  --properties-file FILE   Path to a custom Spark properties file.
                           Default is conf/spark-defaults.conf.

如何传递主节点详细信息以启动工作节点。

Answer 1

工作节点和主节点位于不同的网络中。 一种可能的解决方案是指示必须使用其主机网络的容器（工作节点）

docker run --net=host --name worker1 --env MYMASTER=spark://$HOSTNAME:7077 spkworker