我的Scrapy脚本有问题。我成功登录了我需要抓取的网站，但登录后我在start_urls / login_page上被阻止，而不是解析网址＆＃34; http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&filterGeo=1&activation_status=-1&view=tromb&page=1＆＃34;
我在Stackoverflow上看到了很多关于这个的话题，但是没有一个有同样的问题......
以下是我使用的代码：
import re
from scrapy.spiders.init import InitSpider
from scrapy.http import Request, FormRequest
from items import temaSpiderItem
class temaSpider(InitSpider):
name = 'temaSpider'
allowed_domains = ['http://tm-alumni.eu', 'http://www.tm-alumni.eu/#/annuaire/' ]
start_urls = ['http://www.tm-alumni.eu/']
login_page = 'http://www.tm-alumni.eu/'
directory_page = 'http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&filterGeo=1&activation_status=-1&view=tromb&page=1'
def init_request(self):
return Request(url=self.login_page, callback=self.login)
def login(self, response):
return FormRequest.from_response(response,
formxpath="//form[@id='loginform']",
formdata={'username': 'email', 'password': 'password'},
callback=self.check_login,
dont_filter=True)
def check_login(self, response):
if "My Name" in response.body:
self.log("=========Successfully logged in.=========")
return Request(url=self.directory_page, callback=self.parse_directory, dont_filter=True)
else:
self.log("=========An error in login occurred.=========")
def parse_directory(self, response):
self.log("=========Data is flowing.=========")
self.log(response.url)
以下是我在控制台中找到的内容：
2016-01-05 11:48:49 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2016-01-05 11:48:49 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-05 11:48:49 [scrapy] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'FEED_URI': 'Alumnis.json'}
2016-01-05 11:48:50 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-05 11:48:50 [boto] DEBUG: Retrieving credentials from metadata server.
2016-01-05 11:48:51 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
r = opener.open(req, timeout=timeout)
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Users\nmitchell\AppData\Local\Continuum\Anaconda\lib\urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error timed out>
2016-01-05 11:48:51 [boto] ERROR: Unable to read instance data, giving up
2016-01-05 11:48:51 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddlewar
e, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddlewar
e, ChunkedTransferMiddleware, DownloaderStats
2016-01-05 11:48:51 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthM
iddleware, DepthMiddleware
2016-01-05 11:48:51 [scrapy] INFO: Enabled item pipelines:
2016-01-05 11:48:51 [scrapy] INFO: Spider opened
2016-01-05 11:48:51 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-05 11:48:51 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-05 11:48:52 [scrapy] DEBUG: Crawled (200) <GET http://www.tm-alumni.eu/> (referer: None)
2016-01-05 11:48:53 [scrapy] DEBUG: Crawled (200) <POST http://www.tm-alumni.eu/authentication/index/login> (referer: http://www.tm-
alumni.eu/)
2016-01-05 11:48:53 [temaSpider] DEBUG: =========Successfully logged in.=========
2016-01-05 11:48:55 [scrapy] DEBUG: Crawled (200) <GET http://www.tm-alumni.eu/#/annuaire/diplomes?user_type=1&filterGeo=1&activatio
n_status=-1&view=tromb&page=1> (referer: http://www.tm-alumni.eu/authentication/index/login)
2016-01-05 11:48:55 [temaSpider] DEBUG: =========Data is flowing.=========
2016-01-05 11:48:55 [temaSpider] DEBUG: http://www.tm-alumni.eu/
提前感谢您的帮助！