我无法抓取数据

Question

我正在使用scrapy从网站上抓取数据。这是我的代码

import scrapy


class ShopSpider(scrapy.Spider):
    name = 'shop'
    allowed_domains = ['https://www.shopclues.com/mobiles-smartphones.html?sort_by=bestsellers']
    start_urls = ['http://https://www.shopclues.com/mobiles-smartphones.html?sort_by=bestsellers/']
    custom_settings = {
        'FEED_URI': 'tmp/shop.csv'
    }

    def parse(self, response):

        titles = response.css('img::attr(title)').extract()
        images = response.css('img::attr(data-img)').extract()
        prices = response.css('.p_price::text').extract()
        discounts = response.css('.prd_discount::text').extract()

        for item in zip(titles, prices, images, discounts):
            scraped_info = {
                'title': item[0],
                'price': item[1],
                'image_urls': [item[2]],  # Set's the url for scrapy to download images
                'discount': item[3]
        }

        yield scraped_info

请检查我在哪里做错了？ 另外，我想在滚动时抓取所有数据。因此，它应该处理所有数据，直到我们滚动？那我该怎么办呢？

Answer 1

您遇到以下问题：

1. 错误的allowed_domain（仅需要域）；
2. 断开start_urls（http两次，最后加斜杠）；
3. 错误的意图是在parse函数中产生项目。

在此处检查固定代码：

import scrapy

class ShopSpider(scrapy.Spider):
    name = 'shop'
    allowed_domains = ['shopclues.com']
    start_urls = ['https://www.shopclues.com/mobiles-smartphones.html?sort_by=bestsellers']

    def parse(self, response):
        titles = response.css('img::attr(title)').extract()
        images = response.css('img::attr(data-img)').extract()
        prices = response.css('.p_price::text').extract()
        discounts = response.css('.prd_discount::text').extract()

        for item in zip(titles, prices, images, discounts):
            scraped_info = {
                'title': item[0],
                'price': item[1],
                'image_urls': [item[2]],  # Set's the url for scrapy to download images
                'discount': item[3]
            }

            yield scraped_info