这是我的文件结构
venv/
|-src/
|-gettingstarted/
| |-settings/
| |-__init__.py
| |-base.py
| |-local.py
| |-production.py
|
|-blog/
| |-__init__.py
| |-admin.py
| |-forms.py
| |-models.py
| |-tasks.py
| |-urls.py
| |-views.py
|
|-manage.py
my views.py
from .tasks import add, p_panties
def shopan(request):
# one = scrape_and_store_world()
# two = panties()
# argument_scrapes(one, two)
p_panties.delay()
return redirect('/')
my tasks.py
import requests
import random
import re
import os
from celery import Celery
from bs4 import BeautifulSoup
app = Celery('tasks', backend='redis://localhost', broker='redis://localhost')
@app.task
def add(x, y):
return x + y
@app.task
def reverse(string):
return string[::-1]
@app.task
def p_panties():
def swappo():
user_one = ' "Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" '
user_two = ' "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5)" '
user_thr = ' "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko" '
user_for = ' "Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:10.0) Gecko/20100101 Firefox/10.0" '
agent_list = [user_one, user_two, user_thr, user_for]
a = random.choice(agent_list)
return a
headers = {
"user-agent": swappo(),
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"accept-charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3",
"accept-encoding": "gzip,deflate,sdch",
"accept-language": "en-US,en;q=0.8",
}
pan_url = 'http://www.example.com'
shtml = requests.get(pan_url, headers=headers)
soup = BeautifulSoup(shtml.text, 'html5lib')
video_row = soup.find_all('div', {'class': 'post-start'})
name = 'pan videos'
def youtube_link(url):
youtube_page = requests.get(url, headers=headers)
soupdata = BeautifulSoup(youtube_page.text, 'html5lib')
video_row = soupdata.find_all('p')[0]
entries = [{'text': div,
} for div in video_row]
tubby = str(entries[0]['text'])
urls = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', tubby)
cleaned_url = urls[0].replace('?&autoplay=1', '')
return cleaned_url
def yt_id(code):
the_id = code
youtube_id = the_id.replace('https://www.example.com/embed/', '')
return youtube_id
def strip_hd(hd, move):
str = hd
new_hd = str.replace(move, '')
return new_hd
entries = [{'href': div.a.get('href'),
'text': strip_hd(strip_hd(div.h2.text, '– Official video HD'), '– Oficial video HD').lstrip(),
'embed': youtube_link(div.a.get('href')), #embed
'comments': strip_hd(strip_hd(div.h2.text, '– Official video HD'), '– Oficial video HD').lstrip(),
'src': 'https://i.ytimg.com/vi/' + yt_id(youtube_link(div.a.get('href'))) + '/maxresdefault.jpg', #image
'name': name,
'url': div.a.get('href'),
# 'author': author,
'video': True
} for div in video_row][:13]
return entries
在我的终端
(practice) apples-MBP:blog ray$ celery worker -A tasks -l info
[2016-09-14 11:30:14,537: WARNING/MainProcess] /Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
-------------- celery@apples-MBP.fios-router.home v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Darwin-15.6.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x1037836a0
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost/
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. tasks.add
. tasks.p_panties
. tasks.reverse
[2016-09-14 11:30:14,689: INFO/MainProcess] Connected to redis://localhost:6379//
[2016-09-14 11:30:14,702: INFO/MainProcess] mingle: searching for neighbors
[2016-09-14 11:30:15,710: INFO/MainProcess] mingle: all alone
[2016-09-14 11:30:15,723: WARNING/MainProcess] celery@apples-MBP.fios-router.home ready.
当我尝试运行我的程序时，我得到的错误消息是
[2016-09-14 11:30:32,171: ERROR/MainProcess] Received unregistered task of type 'blog.tasks.p_panties'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see
here了解更多信息。
The full contents of the message body was:
{'timelimit': (None, None), 'task': 'blog.tasks.p_panties', 'chord': None, 'taskset': None, 'errbacks': None, 'id': 'd9814eb1-98a9-4b45-b049-e36ac64fc55c', 'retries': 0, 'args': [], 'utc': True, 'expires': None, 'eta': None, 'kwargs': {}, 'callbacks': None} (263b)
Traceback (most recent call last):
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/worker/consumer.py", line 456, in on_task_received
strategies[name](message, body,
KeyError: 'blog.tasks.p_panties'
我是否应该将其设为应用程序，以便我可以使用它？
我的博客/ celery.py正如你建议我写的那样
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault(
'DJANGO_SETTINGS_MODULE', 'gettingstarted.settings'
)
app = Celery('tasks')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
我如何经营芹菜工人
celery worker -A tasks -l info
我正在博客目录中运行它，因为那是tasks.py所在的位置，否则如果我在src中运行我得到
No module named 'tasks'
但如果我在src中运行它，我会得到
Parent module '' not loaded, cannot perform relative import
答案 0 :(得分：0)
你得到一个很长的警告，告诉你在you're not familiar with it's possible side effects的情况下不要使用 pickle 。所以最好将celery序列化器设置为使用json。
CELERY_TASK_SERIALIZER = 'json'
现在主要问题是，正在运行的任务是，如错误所示， blog.tasks.p_panties ，但是正如运行芹菜工人的输出所示，在芹菜上注册的任务， tasks.p_panties 。
由于您在 tasks.py 中创建了芹菜应用程序实例，因此问题就出现了，因此它正在使用 task。* 这样的名称注册任务。
解决方案是在根目录中创建 celery.py 并在那里设置celery应用程序，然后最终将该app对象导入到tasks.py文件中。
celery.py的内容如下：
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault(
'DJANGO_SETTINGS_MODULE', DEFAULT_SETTINGS_MODULE
)
app = Celery('app_name')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
然后在您的tasks.py中导入，如下所示：
from blog.celery import app
可以看出，在 celery.py 文件中，我实例化了芹菜，并调用了app.config_from_object('django.conf:settings')。这样芹菜就知道从你的django设置文件中读取它的配置参数。所以所有芹菜设置都会去，包括BROKER_URL和其他设置（你直接传递给芹菜对象）
修改
不要像以下那样经营芹菜：celery worker -A tasks -l info
运行它：celery worker -A project_name -l info