Question

我使用 Scrapy 来抓取数据this site。我需要从getlink致电parse。使用yield时，普通呼叫也不起作用，我收到此错误：

2015-11-16 10:12:34 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'generator' in <GET https://www.coldwellbankerhomes.com/fl/miami-dad

E-县/ KVC-17_1,17_3,17_2,17_8 /含-22 /＆GT;

从getlink返回parse函数有效但我需要在返回后执行一些代码。我很困惑，任何帮助都会非常明显。

    # -*- coding: utf-8 -*-

    from scrapy.spiders import BaseSpider
    from scrapy.selector import Selector
    from scrapy.http import Request,Response
    import re
    import csv
    import time

    from selenium import webdriver



    class ColdWellSpider(BaseSpider):
        name = "cwspider"
        allowed_domains = ["coldwellbankerhomes.com"]
        #start_urls = [''.join(row).strip() for row in csv.reader(open("remaining_links.csv"))]
        #start_urls = ['https://www.coldwellbankerhomes.com/fl/boynton-beach/5451-verona-drive-unit-d/pid_9266204/']
        start_urls = ['https://www.coldwellbankerhomes.com/fl/miami-dade-county/kvc-17_1,17_3,17_2,17_8/incl-22/']

        def parse(self,response):

                #browser = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true', '--load-images=false'])
                browser = webdriver.Firefox()
                browser.maximize_window()
                browser.get(response.url)
                time.sleep(5)

                #to extract all the links from a page and send request to those links
                #this works but even after returning i need to execute the while loop
                return self.getlink(response)

                #for clicking the load more button in the page 
                while True:
                    try:
                        browser.find_element_by_class_name('search-results-load-more').find_element_by_tag_name('a').click()
                        time.sleep(3)
                        self.getlink(response)

                    except:
                        break

        def getlink(self,response):
            print 'hhelo'

            c = open('data_getlink.csv', 'a')
            d = csv.writer(c, lineterminator='\n')
            print 'hello2'
            listclass = response.xpath('//div[@class="list-items"]/div[contains(@id,"snapshot")]')

            for l in listclass:
                   link = 'http://www.coldwellbankerhomes.com/'+''.join(l.xpath('./h2/a/@href').extract())

                   d.writerow([link])
                   yield Request(url = str(link),callback=self.parse_link)


       #callback function of Request
       def parse_link(self,response):
                b = open('data_parselink.csv', 'a')
                a = csv.writer(b, lineterminator='\n')
                a.writerow([response.url])

Answer 1

Spider必须返回Request，BaseItem，dict或None，得到＆＃39; generator＆＃39;

getlink()是一个生成器。您正尝试yield生成器parse()。

相反，您可以/应该迭代getlink()调用的结果：

def parse(self, response):
    browser = webdriver.Firefox()
    browser.maximize_window()
    browser.get(response.url)
    time.sleep(5)

    while True:
        try:
            for request in self.getlink(response):
                yield request

            browser.find_element_by_class_name('search-results-load-more').find_element_by_tag_name('a').click()
            time.sleep(3)
        except:
            break

另外，我注意到您同时拥有self.getlink(response)和self.getlink(browser)。后者不会起作用，因为webdriver实例上没有xpath()方法 - 你可能想要从你的webdriver控制的浏览器加载的页面源中make a Scrapy Selector，例如：

selector = scrapy.Selector(text=browser.page_source)
self.getlink(selector)

您还应该查看Explicit Waits with Expected Conditions，而不是通过time.sleep()使用不可靠且缓慢的人工延迟。

另外，我不确定您手动写入CSV而不是使用内置Scrapy Items和Item Exporters的原因是什么。并且，您没有正确关闭文件，也没有使用with()上下文管理器。

此外，尝试捕获更具体的例外和avoid having a bare try/expect block。

python多次返回

1 个答案: