Question

我正在上课的网络刮板。我基本上必须编译网站上的所有http链接并将其写入csv。还需要对它们进行重复数据删除，这就是为什么我要使用一组设备。当它写入csv时，我对所有部分都有完整的期望，整个链接集都写到单行而不是每行一个链接。有人可以查看我的代码并告诉我我所缺少的吗？我在任何地方都找不到解决方法。

我的代码如下：

from bs4 import BeautifulSoup
import requests
import csv
import urllib.parse

base_url = 'https://www.census.gov'

l = set()

r = requests.get("https://www.census.gov/programs-surveys/popest.html")
c = r.content

soup = BeautifulSoup(c, 'html.parser')

file = open('c996webscraper_writer.csv', 'w', newline="")

for link in soup.findAll('a'):
    output = link.get('href')
    abs_url = urllib.parse.urljoin(base_url, output)
    l.add(abs_url)

with file:
    write = csv.writer(file, delimiter = ',', lineterminator = '\r')
    write.writerow(['List of Links'])
    write.writerows([l])

file.close()

这是正在发生的事情的打印输出：

CSV Image

Answer 1

from bs4 import BeautifulSoup
import requests
import csv
import urllib.parse

base_url = 'https://www.census.gov'

l = list()

r = requests.get("https://www.census.gov/programs-surveys/popest.html")
c = r.content

soup = BeautifulSoup(c, 'html.parser')

file = open('c996webscraper_writer.csv', 'w', newline="")

for link in soup.findAll('a'):
    output = link.get('href')
    abs_url = urllib.parse.urljoin(base_url, output)
    l.append(abs_url)

with file:
    write = csv.writer(file)
    write.writerow(['List of Links'])
    for x in l:
        write.writerow([x])

file.close()

CSV编写器将设置写到单行而不是多行

1 个答案: