如何一次从Python脚本运行多个蜘蛛?

时间:2014-05-11 09:52:56

标签: python scrapy-spider

我创建了4只蜘蛛,我想在运行脚本时运行超过1只蜘蛛 到目前为止,我已经尝试过但没有帮助

我有这4只蜘蛛

def BestBuyIphoneA(domain):
    dispatcher.connect(stop_reactor, signal=signals.spider_closed)
    spider = BestBuy(domain=domain)
    settings = get_project_settings()
    settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1}
    crawler = Crawler(settings)
    crawler.signals.connect(spider_closed, signal=signals.spider_closed)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()

def BestBuyLappyA(domain):
    dispatcher.connect(stop_reactor, signal=signals.spider_closed)
    spider = BestBuyLappy(domain=domain)
    settings = get_project_settings()
    settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1}
    crawler2 = Crawler(settings)
    crawler2.signals.connect(spider_closed, signal=signals.spider_closed)
    crawler2.configure()
    crawler2.crawl(spider)
    crawler2.start()

def ShopBotLappyB(domain):
    dispatcher.connect(stop_reactor, signal=signals.spider_closed)
    spider = ShopBotLappy(domain=domain)
    settings = get_project_settings()
    settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1}
    crawler3 = Crawler(settings)
    crawler3.signals.connect(spider_closed, signal=signals.spider_closed)
    crawler3.configure()
    crawler3.crawl(spider)
    crawler3.start()

def ShopBotIphoneB(domain):
    dispatcher.connect(stop_reactor, signal=signals.spider_closed)
    spider = ShopBot(domain=domain)
    settings = get_project_settings()
    settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1}
    crawler4 = Crawler(settings)
    crawler4.signals.connect(spider_closed, signal=signals.spider_closed)
    crawler4.configure()
    crawler4.crawl(spider)
    crawler4.start()

我需要在1个单一命令上运行它们所以我这样做...

BestBuyIphoneA("")
ShopBotIphoneB("")
BestBuyLappyA("")
ShopBotLappyB("")
reactor.run()

这样做的正确方法是什么????

0 个答案:

没有答案