Question

我正在尝试从here下载所有产品的图片。我的蜘蛛看起来像：

from shopclues.items import ImgData
import scrapy
    class multipleImages(scrapy.Spider):
        name='multipleImages'
        start_urls=['http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera',]

        def parse (self, response):
            for url in response.css('div.products-grid div.grid-product):
                yield {
                ImgData(image_urls=[url.css('img::attr(src)').extract()])
                }

和 items.py ：

import scrapy
from scrapy.item import Item
class ShopcluesItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    pass

class ImgData(Item):
    image_urls=scrapy.Field()
    images=scrapy.Field()

但是我在运行蜘蛛时遇到以下错误：

2016-09-29 11:56:19 [scrapy] DEBUG: Crawled (200) <GET http://www.shopclues.com/robots.txt> (referer: None)
2016-09-29 11:56:20 [scrapy] DEBUG: Crawled (200) <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera> (referer: None)
2016-09-29 11:56:20 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'set' in <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera>
2016-09-29 11:56:20 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'set' in <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera>
2016-09-29 11:56:20 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'set' in <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera>
2016-09-29 11:56:20 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'set' in <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera>
2016-09-29 11:56:20 [scrapy] ERROR: Spider must return Request, BaseItem, dict or None, got 'set' in <GET http://www.shopclues.com/electronic-accessories-8/cameras-18/cameras-special.html?search=1&q1=camera>

这个错误意味着什么？可能是错误的原因是什么？

Answer 1

将一个网址列表传递给管道。

 def parse (self, response):
     images = ImgData()
     images['image_urls']=[] 
     for url in response.css('div.products-grid div.grid-product):
         images['image_urls'].append(url.css('img::attr(src)').extract_first())
     yield images

Answer 2

{}是在python或字典中定义集合的表示法。取决于您在大括号内提供的值。如果它是一个列表{a，b，c，d}＆lt; - 这是一个集合，如果它是值{a：b，c：d}＆lt; - 那是一个字典的关键。

您在此行中产生一个集合：

yield {
    ImgData(image_urls=[url.css('img::attr(src)').extract()])
}

我假设你想要一本字典？

yield {
    'images': ImgData(image_urls=[url.css('img::attr(src)').extract()]),
}

蜘蛛必须返回Request，BaseItem，dict或None，得到＆＃39; set＆＃39;

2 个答案: