Question

我想从德国论坛抓取线程。 http://www.musiker-board.de/

实际的不同子论坛位于http://www.musiker-board.de/forum

子论坛：musiker-board.de/forum/subforumname

实际的帖子有这些地址：musiker-board.de/threads/threadname

我想关注所有子论坛的所有链接并提取其中的所有主题，但是线程的URL将不再与起始URL匹配。

但是，如果我选择＆＃34; musiker-board.de /＆＃34;作为起始网址，它不会跟随所有子论坛的链接。

以下是代码：

allowed_domains = ["musiker-board.de"]
start_urls = ['http://www.musiker-board.de/forum/'
             ]
rules = (
         Rule(SgmlLinkExtractor(allow=[r'forum/\w+']), follow=True),
         Rule(SgmlLinkExtractor(allow=[r'threads/\w+']), callback='parse_item'),
         )

def parse_item(self, response):
    #extract items...

我该怎么做才能关注所有musiker-board.de/forum/subforum并提取所有musiker-forum.de/threads/threadname？

Answer 1

以下代码（来自您的代码段）似乎运行正常：

# -*- coding: utf-8 -*-
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class Scrapy1Spider(CrawlSpider):

    name = "musiker"
    allowed_domains = ["musiker-board.de"]
    start_urls = ['http://www.musiker-board.de/forum/'
             ]
    rules = (
        Rule(LinkExtractor(allow=[r'forum/\w+']), follow=True),
        Rule(LinkExtractor(allow=[r'threads/\w+']), callback='parse_item'),
    )

    def parse_item(self, response):
        self.logger.info('response.url=%s' % response.url)

至少有这个输出（截断）：

INFO: response.url=http://www.musiker-board.de/threads/peavey-ms-412-userthread.271458/
INFO: response.url=http://www.musiker-board.de/threads/peavey-5150-6505-etc-userthread.180295/
INFO: response.url=http://www.musiker-board.de/threads/marshall-ma-serie-user-thread.386428/
INFO: response.url=http://www.musiker-board.de/threads/h-k-metal-master-shredder-user-thread.250846/
INFO: response.url=http://www.musiker-board.de/threads/hughes-und-kettner-grandmeister-user-thread.553487/
INFO: response.url=http://www.musiker-board.de/threads/ibanez-userthread.190547/
INFO: response.url=http://www.musiker-board.de/threads/hughes-kettner-edition-blue-user-thread.209499/page-2
INFO: response.url=http://www.musiker-board.de/threads/fender-prosonic-userthread.239519/
INFO: response.url=http://www.musiker-board.de/threads/fender-prosonic-userthread.239519/page-5
INFO: response.url=http://www.musiker-board.de/threads/engl-steve-morse-signature-e656-user-thread.427802/page-2
INFO: response.url=http://www.musiker-board.de/threads/engl-sovereign-user-thread.136266/page-20
INFO: response.url=http://www.musiker-board.de/threads/engl-steve-morse-signature-e656-user-thread.427802/
INFO: response.url=http://www.musiker-board.de/threads/engl-sovereign-user-thread.136266/page-19
INFO: response.url=http://www.musiker-board.de/threads/engl-sovereign-user-thread.136266/page-18
INFO: response.url=http://www.musiker-board.de/threads/engl-invader-user-thread.248090/page-5
INFO: response.url=http://www.musiker-board.de/threads/engl-sovereign-user-thread.136266/
INFO: response.url=http://www.musiker-board.de/threads/engl-invader-user-thread.248090/page-4
INFO: response.url=http://www.musiker-board.de/threads/engl-invader-user-thread.248090/page-3
INFO: response.url=http://www.musiker-board.de/threads/fender-cybertwin-userthread.305789/
INFO: response.url=http://www.musiker-board.de/threads/fenders-famose-farbwelten.454766/

Scrapy：跟随正则表达式的链接

1 个答案: