芹菜在长时间运行的任务执行中引发WorkerShutdown和WorkerLostError

时间:2018-04-26 07:01:53

标签: linux process celery

我正在使用芹菜来运行正在侦听某些Web连接并记录消息的任务。环境信息如下:

software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.6.3 billiard:3.5.0.3 
py-amqp:2.2.2 platform -> system:Linux arch:64bit imp:CPython loader -> 
celery.loaders.app.AppLoader settings -> transport:amqp results:disabled

这些任务应该始终像我在async函数中使用true循环时一样运行,以保持Web服务器发送的记录消息。然而,几个小时后,所有芹菜任务都停止了。我发现我的芹菜工人的主要过程和子过程仍然在运行,但是子过程的开始时间比主要过程要新得多。

PID STARTED ELAPSED COMMAND
15785 Mon Apr 23 03:20:01 2018 2-04:53:38 /home/ubuntu/anaconda3/bin/python /home/ubuntu/anaconda3/bin/celery worker -A data_fetcher --loglevel=info --logfile=log.txt --concurrency=10
18587 Tue Apr 24 08:04:20 2018 1-00:09:19 /home/ubuntu/anaconda3/bin/python /home/ubuntu/anaconda3/bin/celery worker -A data_fetcher --loglevel=info --logfile=log.txt --concurrency=10
18588 Tue Apr 24 08:04:20 2018 1-00:09:19 /home/ubuntu/anaconda3/bin/python /home/ubuntu/anaconda3/bin/celery worker -A data_fetcher --loglevel=info --logfile=log.txt --concurrency=10

芹菜日志文件中的更多相关信息(所有子进程):

[2018-04-24 08:04:14,623: ERROR/MainProcess] Process 'ForkPoolWorker-6' pid:15794 exited with 'exitcode 70'
[2018-04-24 08:04:14,633: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 70.',)
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start
return self.obj.start()
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 320, in start
blueprint.start(self)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/celery/worker/loops.py", line 77, in asynloop
raise WorkerShutdown(should_stop)
celery.exceptions.WorkerShutdown: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 70.

为什么子进程关闭并重新启动? (芹菜的任务再也没有排队了。) 保持这些长期运行的任务永远运行的最佳做法是什么? 任何帮助将不胜感激!

0 个答案:

没有答案