Question

我想在搜索中抓取一个单词列表，但它只抓取第一个单词。关键字列表就像每行一个字。

这是我的代码的一部分：

...
def parse(self, response):
...
def after_login(self, response):
...


def start_request(self, response):

    with open('keywords.txt', 'rt') as f:
        for keys in f.readlines():
                query = keys.strip()
                key = {'keywords': query}
                results = FormRequest.from_response(response,formdata=key,callback=self.parse_pa$
                yield results

def parse_page(self,response):
...

如果我将yield结果放在for循环之外，只给我最后一个关键字。（这对于获取最后一个关键字是有意义的）我几次改变了for循环代码。

提前谢谢

更新：

这就是我得到的：

INFO: Spider opened
[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
[scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6026
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.advertcn.com/search.php?mod=forum> (referer: None)
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.advertcn.com/search.php?mod=forum&searchid=313&orderby=lastpost&ascdesc=desc&searchsubmit=yes&kw=AdCombo> from <POST http://www.advertcn.com/search.php?mod=forum>
[scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.advertcn.com/search.php?mod=forum&searchid=313&orderby=lastpost&ascdesc=desc&searchsubmit=yes&kw=AdCombo> (referer: http://www.advertcn.com/search.php?mod=forum)
...
"It crawls the first word"
...
[scrapy.core.engine] INFO: Closing spider (finished)
[scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1268,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 2,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 9093,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/302': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(...),
 'item_scraped_count': 25,
 'log_count/DEBUG': 29,
 'log_count/INFO': 7,
 'log_count/WARNING': 1,
 'memusage/max': 50585600,
 'memusage/startup': 50585600,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(...)}
 [scrapy.core.engine] INFO: Spider closed (finished)

所以它只抓取txt文件中的第一个单词... doenst抓取任何其他单词

scrapy formrequest从文本文件中获取关键字

0 个答案: