Question

import requests
import urllib2
from bs4 import BeautifulSoup
from pprint import pprint
import pandas as pd
import bs4

url = 'https://www.namus.gov/MissingPersons/Case#/53061'
page = urllib2.urlopen(url)
soup = BeautifulSoup(page, 'html')
#print(soup.prettify())

findall = soup.find_all("a")

for link in findall:
    pprint(link.get("ng-href"))

运行代码时，我设置了一个元组而不是链接。我试过引用href，src，ng-href和non work。我只有在确实需要以字符串形式链接到Google地图时，才能拉出subSection。

#I get this: u'{{subSection.mapLink()}}'
#when I really need this: #"http://www.google.com/maps/place/35.9467011,-84.03260329999999"

我尝试抓取的实际字符串如下：

<a ng-if="subSection.mapLink()" class="icon-text-link" ng-href="http://www.google.com/maps/place/35.9467011,-84.03260329999999" target="_blank" href="http://www.google.com/maps/place/35.9467011,-84.03260329999999">
                <i class="icon-location-pin"></i><span>Map</span>
            </a>

Answer 1

由于这是一个有角度的网站，因此有很多信息是使用Javascript动态加载的，您可以检查“网络”标签以了解从何处检索这些数据。在这种情况下，这是具有以下模板的JSON API：

https://www.namus.gov/api/CaseSets/NamUs/MissingPersons/Cases/{CASE_ID}

它为您提供嵌入在页面中的所有信息，地图URL可以动态生成，例如：

import requests

id = '53061'
resp = requests.get('https://www.namus.gov/api/CaseSets/NamUs/MissingPersons/Cases/{}'.format(id))
body = resp.json()

loc = body['sighting']['publicGeolocation']
coord = loc['coordinates']
print("address         : " + loc['formattedAddress'])
print("google map link : " + "http://www.google.com/maps/place/{},{}".format(coord['lat'],coord['lon']))

通过网络抓取从嵌入式Google Map链接中提取坐标

1 个答案: