Question

我正在尝试抓取一个页面并从我抓取的页面中获取一个电话号码，是的，我已经引用了beautifulsoup文档，只需知道如何抓取页面并从该页面获取信息，任何建议？

这是代码

Main.py

#!/usr/bin/env python
# -*- coding: utf-8 -*- 

import requests
import urllib 
from bs4 import BeautifulSoup


glimit = 100


def my_spider(max_pages):
    page = 2
    while page <= max_pages:
        url = 'http://www.bbb.org/search/?type=name&input=constrution&location=Austin%2c+TX&filter=combined&accredited=&radius=5000&country=USA&language=en&codeType=YPPA'
        url_2 = url + '&page='+ str(page) +'&source=bbbse' 
        source_code = requests.get(url_2)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html5lib")
        limit = glimit
        li = soup.find('h4', {'class': 'hcolor'})
        children = li.find_all("a")
        for result in children:
            href = "http://www.bbb.org" + result.get('href')
            owl = (result.string)
            print owl
            get_single_item_data(href)
        page += 1



def get_single_item_data(item_url):
    source_code = requests.get(item_url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text, "html5lib")
    limit = glimit
    mysoup = soup.findAll('h3',{'class': 'address__heading' })[:limit]
    mysoup2 = mysoup.find_all("a")
    for item in mysoup2:
        href = "http://www.bbb.org" + item.get('href')
        print (item.string)


my_spider(2)

这是错误

Traceback (most recent call last):
  File "main.py", line 44, in <module>
    my_spider(2)
  File "main.py", line 27, in my_spider
    get_single_item_data(href)
  File "main.py", line 33, in get_single_item_data
    source_code = requests.get(item_url)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
    raise LocationParseError("Failed to parse: %s" % url)
urllib3.exceptions.LocationParseError: Failed to parse: Failed to parse: www.bbb.orghttp:

Answer 1

您的代码中存在各种问题。

1）您不需要href = "http://www.bbb.org" +。删除"http://www.bbb.org"，因为链接已经拥有主持人。

2）

mysoup = soup.findAll('h3',{'class': 'address__heading' })[:limit]
mysoup2 = mysoup.find_all("a")

您正试图在列表中找到a个标签。您必须重复mysoup或使用find代替findAll。

我已更新您的代码。找到它here。

美丽的汤没有解析

1 个答案: