我正在构建一个抓取应用程序,该应用程序将浏览多个页面,从这些页面抓取并返回结果。我使用Redis排队任务。我碰到了一个结果问题。当我运行脚本并仅从一页抓取数据时,redis成功地将结果存储了500秒(默认情况下),并且能够在job_id下检索它们。但是,当我尝试刮取多个页面时,redis返回None作为结果。返回的结果将由函数正常打印出来。我将在下面发布代码,也许现在是您为什么会这样:
contacts = []
def flatten(list):
flattened = []
for i in list:
for z in i:
flattened.append(z)
return flattened
def start(url, cookie=True):
if cookie == True:
load_cookie(driver)
driver.get(url)
start_scroll(driver)
#checking if the page has contacts
elements = driver.find_elements_by_xpath("some xml")
if len(elements) > 0:
profiles = get_data(driver)
contacts.append(profiles)
next_page_url = next_url(driver)
print(profiles)
start(next_page_url, False)
else:
print('returenter code herening contacts')
flattened = flatten(contacts)
print(flattened)
return flattened