Question

python新手并试图下载NIST NVD JSON文件。我尝试了几种方法，但它只写了大约324字节的文件。如果我做了一个实际上工作的文件，但有几个文件要下载。

我确实尝试调整chunk_size但仍然无法获得1到6mb的zip文件下载

from requests import get

def download(url, filename):
    response = get(url, stream = True)
    with open(filename, "wb") as file:
        for chunk in response.iter_content(chunk_size=1024):
            if chunk:
                file.write(chunk)
    print('Downloaded! ', filename)

with open('NVD_JSON_SOURCE_URLS.txt') as f:
    for line in f:    
        filename = line.split('/')[-1]
        url = line
        download(url, filename)

输入有效并开始下载，只是永远不会完成它们。很明显，我在这里失去了一些令人沮丧的简单但在2天后我没有接近。感谢。

Answer 1

我觉得在Python中像这样的实例会让人感到痛苦。这种方法经常对我有用：

import requests
import shutil

def download_file(url, filename):
    r = requests.get(url, stream=True)
    with open(filename, 'wb') as f:
        shutil.copyfileobj(r.raw, f)

这将整个文件流式传输到内存然后写入。所以不会为大文件工作，但你只是谈论几个MB应该工作正常。

Answer 2

我认为line有一些空白字符，因此如果您使用line从strip()中删除空白字符，则代码应该有效。

for line in f:    
    line = line.strip()
    ...

我测试了它，它适用于我。

Answer 3

因为当您从.txt文件中读取数据时，行末尾会有换行符。所以你应该在第一个中删除换行符。

from requests import get

def download(url, filename):
    response = get(url, stream = True)
    with open(filename, "wb") as file:
        for chunk in response.iter_content(chunk_size=1024):
            if chunk:
                file.write(chunk)
    print('Downloaded! ', filename)

with open('NVD_JSON_SOURCE_URLS.txt') as f:
    for line in f:
        line = line.strip()    
        filename = line.split('/')[-1]
        url = line
        download(url, filename)

Python文件下载失败

3 个答案: