Question

没有图书馆......

我尝试从网页上获取所有链接标题，代码如下

url="http://einstein.biz/"
m = urllib.request.urlopen(url)
msg = m.read()
titleregex=re.compile('<a\s*href=[\'|"].*?[\'"].*?>(.+?)</a>')
titles = titleregex.findall(str(msg))
print(titles)

标题是

['Photo Gallery', 'Bio', 'Quotes', 'Links', 'Contact', 'official store', '\\xe6\\x97\\xa5\\xe6\\x9c\\xac\\xe8\\xaa\\x9e', '<img\\n\\t\\tsrc="http://corbisrightsceleb.122.2O7.net/b/ss/corbisrightsceleb/1/H.14--NS/0"\\n\\t\\theight="1" width="1" border="0" alt="" />']

这不理想，我想只有如下：

['Photo Gallery', 'Bio', 'Quotes', 'Links', 'Contact', 'official store']

如何修改我的代码？

Answer 1

处理html或xml文件时必须使用BeautifulSoup。

>>> url="http://einstein.biz/"
>>> import urllib.request
>>> m = urllib.request.urlopen(url)
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(m)
>>> s = soup.find_all('a')
>>> [i.string for i in s]
['Photo Gallery', 'Bio', 'Quotes', 'Links', 'Contact', 'official store', '日本語', None]

<强>更新

>>> import urllib.request
>>> url="http://einstein.biz/"
>>> m = urllib.request.urlopen(url)
>>> msg = m.read()
>>> regex = re.compile(r'(?s)<a\s*href=[\'"].*?[\'"][^<>]*>([A-Za-z][^<>]*)</a>')
>>> titles = regex.findall(str(msg))
>>> print(titles)
['Photo Gallery', 'Bio', 'Quotes', 'Links', 'Contact', 'official store']

Answer 2

我肯定会像@serge提到的那样关注BeautifulSoup。为了使它更具说服力，我已经包含了完全符合您需要的代码。

from bs4 import BeautifulSoup
soup = BeautifulSoup(msg)          #Feed BeautifulSoup your html.
for link in soup.find_all('a'):    #Look at all the 'a' tags.
    print(link.string)             #Print out the descriptions.

返回

Photo Gallery
Bio
Quotes
Links
Contact
official store

Answer 3

我比BeautifulSoup更喜欢lxml.html，它支持xpath和cssselector。

import requests
import lxml.html

res = requests.get("http://einstein.biz/")
doc = lxml.html.fromstring(res.content)
links = doc.cssselect("a")
for l in links:
    print l.text

如何从html页面中提取所有链接名称

3 个答案: