Question

我正在尝试从eshop网站返回html作为字符串，但是又找回了一些奇怪的字符。当我查看网络控制台时，我在html中看不到这些字符。当在jupyter笔记本的pandas数据框中显示html时，我也看不到这些字符。链接为https://www.powerhousefilms.co.uk/collections/limited-editions/products/immaculate-conception-le。我在本网站上的另一产品上也使用了相同的方法，但是在此页面上只能看到这些字符。该网站中的其他页面没有此问题。

html = requests.get(url).text
soup = BeautifulSoup(html)
elem = soup.find_all('div', {'class': product-single_description rte'})
s = str(elem[0])

s看起来像：

    <div class="product-single__description rte">
<div class="product_description">
<div>
<div>
<div><span style="color: #000000;"><em>THIS ITEM IS AVAILABLE TO PRE-ORDER.Â PLEASE NOTEÂ THAT YOUR PAYMENTÂ WILL BEÂ TAKEN IMMEDIATELY, AND THATÂ THE ITEMÂ WILL BE DISPATCHED JUSTÂ BEFORE THE LISTEDÂ RELEASEÂ DATE.Â </em></span></div>
<div><span style="color: #000000;"><em>Â </em></span></div>
<div><span style="color: #000000;"><em>SHOULD YOU ORDER ANY OF THEÂ ALREADY RELEASED ITEMS FROM OURÂ CATALOGUE AT THE SAME TIME AS THIS PRE-ORDER ITEM, PLEASE NOTE THATÂ YOUR PURCHASES WILL ALL BE SHIPPED TOGETHER WHENÂ THIS PRE-ORDERÂ ITEM BECOMES AVAILABLE.</em></span></div>
</div>
<div><span style="color: #38761d;">Â </span></div>
<div>
<strong>(Jamil Dehlavi, 1992)</strong><br/><em>Release date:Â 25 March 2019</em><br/>Limited Blu-ray Edition (World Blu-ray premiere)<br/><br/>A Western couple (played by Melissa Leo and James Wilby) working in Pakistan visit an unconventional holy shrine to harness its spiritual powers to help them conceive a child. They are lavished with the attentions of the shrineâ€™s leader (an exceptional performance from Zia Mohyeddin â€“ <em>Lawrence of Arabia</em>, <em>Khartoum</em>) and her followers, but their methods and motives are not all that they seem, and the coupleâ€™s lives are plunged into darkness.<br/><br/>This ravishing, unsettling film from director Jamil Dehlavi (<em>The Blood of Hussain</em>, <em>Born of Fire</em>) is a deeply personal work which raises questions of cultural and sexual identity, religious fanaticism and the abuses of power. The brand-new 2K restoration from the original negative was supervised and approved by Dehlavi and cinematographer Nic Knowland.<br/><br/><strong>INDICATOR LIMITED EDITION BLU-RAY SPECIAL FEATURES:</strong>
</div>
<div>
<ul>
<li>New 2K restoration by Powerhouse Films from the original negative, supervised and approved by director Jamil Dehlavi and cinematographer Nic Knowland</li>
<li>
<div>Original stereo audio</div>
</li>
<li>
<div>Alternative original mono mix</div>

我尝试指定编码，但仍然得到奇怪的字符。对于该网站上的50多种产品，只有少数产品存在此问题。

我的抓取方式是否存在问题，或者是否可能是清除此问题的简便方法。

谢谢

Answer 1

使用这段代码来下载网页中的可见内容。只需在page_url

中输入url

from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request
import os


page_url = "URL Here"
def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
        return False
    if isinstance(element, Comment):
        return False
    return True


def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)
    return u" ".join(t.strip() for t in visible_texts)

def Extract_Text(html_bytes, url):
    text_data = text_from_html(html_bytes)
    f = open("DOC.txt", "w")
    string = str(url) + "\n" + text_data
    f.write(str(string))
    f.close()

html_string = ''
response = urlopen(page_url)
if 'text/html' in response.getheader('Content-Type'):
    html_bytes = response.read()
    html_string = html_bytes.decode("utf-8")
Extract_Text(html_bytes, page_url)

Answer 2

所以事实证明excel是造成这种情况的原因。当我保存到CSV并在excel中打开时，我得到了奇怪的结果。

为防止这种情况，我使用了df.to_csv('df.csv', index=False, encoding = 'utf-8-sig')。指定编码摆脱了奇怪的字符。

Python Writing Weird Unicode to CSV提供了有关随后编码以及excel如何渗透csv文件的一些信息。

使用Beautiful Soup进行网页抓取时出现奇怪的字符

2 个答案: