使用Supervisor和Django在Elastic Beanstalk上运行Celery作为守护进程

时间：2017-05-30 17:36:17

标签： python django celery elastic-beanstalk supervisord

我试图让Celery在Django的EB环境中运行，我已经非常接近让它全部运行。我有这个主管的配置文件

#!/usr/bin/env bash

# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}

# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv,USER="ec2-user"

[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# Reread the supervisord config
echo reread
supervisorctl -c /opt/python/etc/supervisord.conf reread

echo reload
# supervisorctl -c /opt/python/etc/supervisord.conf reload

# Update supervisord in cache without restarting all services
echo update
supervisorctl -c /opt/python/etc/supervisord.conf update

# Start/Restart celeryd through supervisord
echo restart
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker

现在，当我部署它时，celery beat会运行（通过sshing并使用ps -aux |来验证正在运行的进程），但没有celeryd-worker。在我的activity.log中，我有这个。

[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-05-30T17:23:37.675Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

  directory=/opt/python/current/app
  user=ec2-user
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  reread
  No config updates to processes
  reload
  update
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)

所以当我尝试启动工作人员而不是打败工作人员时会出现问题。

如果我通过SSH进入EB并查看/var/log/celery-worker.log文件，我会看到这一点。

[2017-05-30 12:30:21,370: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/hub.py", line 282, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 139, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 315, in _get_async
    q, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 412, in asynsqs
    AsyncSQSConnection, _asynsqs.regions(),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 400, in _aws_connect_to
    port=port)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/sqs/connection.py", line 39, in __init__
    https_connection_factory=https_connection_factory, **kwargs
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 223, in __init__
    AsyncConnection.__init__(self, http_client, **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 172, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

我在本地计算机上执行此操作时也遇到过这种情况。虽然我确保使用带有此容器命令的nss安装curl。

container_commands:
  01_libcurl_ssl_backend:
    command: "export PYCURL_SSL_LIBRARY=nss"
    leader_only: true

pycurl正在寻找什么。我的需求文件中也有pycurl，如果我冻结，我可以看到安装了pycurl。我通过重新创建我的虚拟环境在本地解决了这个问题。但不太确定该怎么做。

修改1 我加入了

PYCURL_SSL_LIBRARY: nss

到我的配置文件中的选项设置，并且能够部署并显示略有不同的行为。在查看活动日志之后。

而不是

update
restart
celeryd-beat: stopped
celeryd-beat: started
celeryd-worker: ERROR (not running)
celeryd-worker: ERROR (abnormal termination)

我得到了：

    reread
  celeryd-beat: changed
  celeryd-worker: changed
  reload
  update
  celeryd-beat: stopped
  celeryd-beat: updated process group
  celeryd-worker: stopped
  celeryd-worker: updated process group
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: stopped
  celeryd-worker: ERROR (abnormal termination)

但是celery-worker.log在pycurl

中仍然有相同的错误

2 个答案:

答案 0 :(得分：3)

1 - 按照这个问题的答案：https://stackoverflow.com/a/41097851/3407986

2 - 您必须在包eb配置文件/部分中添加libcurl-devel，请参阅我的。 ebextensions / 02_packages.config 文件：

packages:
  yum:
    // ... your another packages where ...
    // add libcurl-devel, dependency of pycurl package
    libcurl-devel: []

答案 1 :(得分：1)

使用与用于运行应用程序的EB相同的python，尝试使用SSH到您的实例并自己导入pycurl。然后你会看到实际的错误。

例如，这是我在没有--global-option="--with-nss"的情况下构建pycurl时得到的结果：

[root@ip-172-31-2-149 log]# /opt/python/run/venv/local/bin/python Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. \>>> import pycurl Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)

相关问题
最新问题