Question

我一直在抓取此网站并尝试存储属性，而某些属性确实被抓取了，而有些只是被抓取而未抓取：

class CapeWaterfrontSpider(scrapy.Spider):
    name = "cape_waterfront"
    start_urls = ['https://www.capewaterfrontestates.co.za/template/Properties.vm/listingtype/SALES']

    def parse(self, response):
        for prop in response.css('div.col-sm-6.col-md-12.grid-sizer.grid-item'):

            link = prop.css('div.property-image a::attr(href)').get()

            bedrooms = prop.css('div.property-details li.bedrooms::text').getall()
            bathrooms = prop.css('div.property-details li.bathrooms::text').getall()
            gar = prop.css('div.property-details li.garages::text').getall()

            if len(bedrooms) == 0:
                bedrooms.append(None)
            else:
                bedrooms = bedrooms[1].split()
            if len(bathrooms) == 0:
                bathrooms.append(None)
            else:
                bathrooms = bathrooms[1].split()
            if len(gar) == 0:
                gar.append(None)
            else:
                gar = gar[1].split()

            yield scrapy.Request(
                link,
                meta={'item': {
                    'agency': self.name,
                    'url': link,
                    'title': ' '.join(prop.css('div.property-details p.intro::text').get().split()),
                    'price': ''.join(prop.css('div.property-details p.price::text').get().split()),
                    'bedrooms': str(bedrooms),
                    'bathroom':  str(bathrooms),
                    'garages': str(gar)
                }},
                callback=self.get_loc,
            )

        next_page = response.css('p.form-control-static.pagination-link a::attr(href)').get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

任何建议如何使这项工作？预先非常感谢

Answer 1

定义选择器的方式容易出错。此外，几乎没有故障的选择器很少。指向下一页的链接也无法正常工作。它只会转到第1页，然后退出。最后，我不知道css选择器中{ "sample":{ "testString":"foo" } }的用法，因此我不得不以某种尴尬的方式来挖掘下一个同级对象。

next_sibling

如果您想采用一种更清洁的方法来获得这三个项目，我想class CapeWaterfrontSpider(scrapy.Spider): name = "cape_waterfront" start_urls = ['https://www.capewaterfrontestates.co.za/template/Properties.vm/listingtype/SALES'] def parse(self, response): for prop in response.css('.grid-item'): link = prop.css('.property-image a::attr(href)').get() bedrooms = [elem.strip() for elem in prop.css(".bedrooms::text").getall()] bedrooms = bedrooms[-2] if len(bedrooms)>=1 else None bathrooms = [elem.strip() for elem in prop.css(".bathrooms::text").getall()] bathrooms = bathrooms[-2] if len(bathrooms)>=1 else None gar = [elem.strip() for elem in prop.css(".garages::text").getall()] gar = gar[-2] if len(gar)>=1 else None yield scrapy.Request( link, meta={'item': { 'agency': self.name, 'url': link, 'bedrooms': bedrooms, 'bathroom': bathrooms, 'garages': gar }}, callback=self.get_loc, ) next_page = response.css('.pagination-link a.next::attr(href)').get() if next_page: next_page = response.urljoin(next_page) yield scrapy.Request(next_page, callback=self.parse) def get_loc(self,response): items = response.meta['item'] print(items)是您要坚持的目标：

xpath

为简洁起见，我已经淘汰了两三个字段，我想您可以对其进行管理。

网站被抓取但未被抓取Scrapy

1 个答案: