Question

我想知道为什么我的Link Extractor没有得到任何链接。我正在扩展CrawlSpider并在我的链接提取器中设置tags和attrs属性。我正在尝试从网页上的选择框中删除链接（http://www.realclearpolitics.com/epolls/latest_polls/president/），所以我正在寻找带有值属性的选项标签。

在此处查看代码：https://github.com/jtfairbank/RCP-Poll-Scraper/blob/master/polldata/spiders/pres.py

这是根目录：https://github.com/jtfairbank/RCP-Poll-Scraper

有什么想法吗？谢谢！

Answer 1

您的链接提取器未获取任何链接，因为您正在使用<option>搜索tags=['option']个标记，并且没有选择或选项标记在this page。

尝试restrict_xpaths='//td[@class="lp-race"]'：

>>> SgmlLinkExtractor(
...     allow=(r"epolls/2012/president/[a-z]{2}/[a-z]+_romney_vs_obama-[0-9]{4}\.html"),
...     allow_domains=('realclearpolitics.com',),
...     restrict_xpaths='//td[@class="lp-race"]',
... ).extract_links(response)

[Link(url='http://www.realclearpolitics.com/epolls/2012/president/fl/florida_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/oh/ohio_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/va/virginia_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ia/iowa_romney_vs_obama...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wi/wisconsin_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/or/oregon_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mo/missouri_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ca/california_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/md/maryland_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mi/michigan_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/co/colorado_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mn/minnesota_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/wa/washington_romney_vs...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ct/connecticut_romney_v...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/nv/nevada_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/mt/montana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/pa/pennsylvania_romney_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/in/indiana_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ma/massachusetts_romney...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ga/georgia_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/id/idaho_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/az/arizona_romney_vs_ob...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/me/maine_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/la/louisiana_romney_vs_...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/hi/hawaii_romney_vs_oba...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/tx/texas_romney_vs_obam...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ar/arkansas_romney_vs_o...',
 Link(url='http://www.realclearpolitics.com/epolls/2012/president/ne/nebraska_romney_vs_o...',

Scrapy SGML Link Extractor带有自定义标签和属性

1 个答案: