使用请求模块向NASA网站提出请求

时间：2020-06-23 14:28:45

标签： python python-requests http-headers

我想使用Python中的请求模块从Nasa网站下载Apollo图片。

这是我的起始代码-

tf.data.Dataset

但是，该网站可以检测到我没有使用我的浏览器。 我也尝试了其他标头，但是它不起作用 我要怎么做才能超越这个？ 请告诉

2 个答案:

答案 0 :(得分：3)

使用https://images.nasa.gov/search-results?q=Apollo模块向requests发出HTTP GET请求时，您可以准确地从服务器获取所需的资源：该HTML文件。没有其他的。当您使用浏览器发出相同的请求时，浏览器知道会解析该HTML文件以执行脚本，发出Ajax请求/ XHR请求，基于HTML中的元数据/链接/样式表发出请求，等等。浏览器以人眼可以看到的方式完全呈现给定页面。当您对页面进行简单的HTTP GET请求时（如您所做的那样），您只需返回准系统模板，即尚未填充DOM的系统，因为这通常意味着在以后的某个时间异步发生。时间。

您仍然可以使用requests模块，但使用方式有所不同：您无需向网页发出HTTP GET请求，而是向NASA RESTful图像资产API发出HTTP GET请求。填充页面时，浏览器会向其请求的API。这是理想的选择，因为您不必使用BeautifulSoup或类似方法刮取页面，只需模仿浏览器发出的相同请求，然后返回JSON：

def main():

    import requests

    url = "https://images-api.nasa.gov/search"

    params = {
        "q": "apollo",
        "page": "1",
        "media_type": "image",
        "year_start": "1920",
        "year_end": "2020"
    }

    response = requests.get(url, params=params)
    response.raise_for_status()

    images = response.json()["collection"]["items"]
    for image in images:
        thumbnail_url = image["links"][0]["href"]
        image_url = thumbnail_url[:thumbnail_url.rfind("~")] + "~orig.jpg"
        print(image_url)

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

输出：

https://images-assets.nasa.gov/image/APOLLO 50th_FULL COLOR_300DPI/APOLLO 50th_FULL COLOR_300DPI~orig.jpg
https://images-assets.nasa.gov/image/s69-35505/s69-35505~orig.jpg
https://images-assets.nasa.gov/image/S69-35503/S69-35503~orig.jpg
https://images-assets.nasa.gov/image/7995383/7995383~orig.jpg
https://images-assets.nasa.gov/image/S69-16402/S69-16402~orig.jpg
https://images-assets.nasa.gov/image/S69-34968/S69-34968~orig.jpg
https://images-assets.nasa.gov/image/S69-34969/S69-34969~orig.jpg
https://images-assets.nasa.gov/image/PIA13058/PIA13058~orig.jpg
https://images-assets.nasa.gov/image/PIA14003/PIA14003~orig.jpg
https://images-assets.nasa.gov/image/S69-60662/S69-60662~orig.jpg
https://images-assets.nasa.gov/image/S69-34313/S69-34313~orig.jpg
https://images-assets.nasa.gov/image/S69-33999/S69-33999~orig.jpg
https://images-assets.nasa.gov/image/S69-27915/S69-27915~orig.jpg
https://images-assets.nasa.gov/image/S68-56002/S68-56002~orig.jpg
https://images-assets.nasa.gov/image/200907200097HQ/200907200097HQ~orig.jpg
https://images-assets.nasa.gov/image/S68-26890/S68-26890~orig.jpg
https://images-assets.nasa.gov/image/S68-26958/S68-26958~orig.jpg
https://images-assets.nasa.gov/image/S68-27041/S68-27041~orig.jpg
https://images-assets.nasa.gov/image/S68-27076/S68-27076~orig.jpg
https://images-assets.nasa.gov/image/S68-27161/S68-27161~orig.jpg
https://images-assets.nasa.gov/image/S68-26936/S68-26936~orig.jpg
https://images-assets.nasa.gov/image/S69-27089/S69-27089~orig.jpg
https://images-assets.nasa.gov/image/200907200096HQ/200907200096HQ~orig.jpg
https://images-assets.nasa.gov/image/200907200094HQ/200907200094HQ~orig.jpg
https://images-assets.nasa.gov/image/200907200095HQ/200907200095HQ~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110018/NHQ201812110018~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110017/NHQ201812110017~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110027/NHQ201812110027~orig.jpg
https://images-assets.nasa.gov/image/S69-25478/S69-25478~orig.jpg
https://images-assets.nasa.gov/image/S69-25861/S69-25861~orig.jpg
https://images-assets.nasa.gov/image/S69-25862/S69-25862~orig.jpg
https://images-assets.nasa.gov/image/S69-25881/S69-25881~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0104/KSC-20190716-PH_FWM04_0104~orig.jpg
https://images-assets.nasa.gov/image/200907200098HQ/200907200098HQ~orig.jpg
https://images-assets.nasa.gov/image/S67-50903/S67-50903~orig.jpg
https://images-assets.nasa.gov/image/S69-19983/S69-19983~orig.jpg
https://images-assets.nasa.gov/image/as17-162-24050/as17-162-24050~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110008/NHQ201812110008~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110019/NHQ201812110019~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110042/NHQ201812110042~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110009/NHQ201812110009~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110010/NHQ201812110010~orig.jpg
https://images-assets.nasa.gov/image/S68-26668/S68-26668~orig.jpg
https://images-assets.nasa.gov/image/Maura_test_s69-36593/Maura_test_s69-36593~orig.jpg
https://images-assets.nasa.gov/image/S69-36593/S69-36593~orig.jpg
https://images-assets.nasa.gov/image/200907200015HQ/200907200015HQ~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0123/KSC-20190716-PH_FWM04_0123~orig.jpg
https://images-assets.nasa.gov/image/as09-23-3596/as09-23-3596~orig.jpg
https://images-assets.nasa.gov/image/S67-49447/S67-49447~orig.jpg
https://images-assets.nasa.gov/image/S69-35099/S69-35099~orig.jpg
https://images-assets.nasa.gov/image/S69-26148/S69-26148~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_KLS02_0062/KSC-20190716-PH_KLS02_0062~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0063/KSC-20190716-PH_FWM04_0063~orig.jpg
https://images-assets.nasa.gov/image/jsc2007e034221/jsc2007e034221~orig.jpg
https://images-assets.nasa.gov/image/9309375/9309375~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110004/NHQ201812110004~orig.jpg
https://images-assets.nasa.gov/image/0102627/0102627~orig.jpg
https://images-assets.nasa.gov/image/200907200091HQ/200907200091HQ~orig.jpg
https://images-assets.nasa.gov/image/200907200093HQ/200907200093HQ~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110006/NHQ201812110006~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110007/NHQ201812110007~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110041/NHQ201812110041~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110034/NHQ201812110034~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110036/NHQ201812110036~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110037/NHQ201812110037~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110030/NHQ201812110030~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110040/NHQ201812110040~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110016/NHQ201812110016~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110002/NHQ201812110002~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110025/NHQ201812110025~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110035/NHQ201812110035~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110014/NHQ201812110014~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110015/NHQ201812110015~orig.jpg
https://images-assets.nasa.gov/image/NHQ201812110038/NHQ201812110038~orig.jpg
https://images-assets.nasa.gov/image/S69-35507/S69-35507~orig.jpg
https://images-assets.nasa.gov/image/S69-35504/S69-35504~orig.jpg
https://images-assets.nasa.gov/image/S71-37963/S71-37963~orig.jpg
https://images-assets.nasa.gov/image/as17-152-23272/as17-152-23272~orig.jpg
https://images-assets.nasa.gov/image/0101493/0101493~orig.jpg
https://images-assets.nasa.gov/image/6761216/6761216~orig.jpg
https://images-assets.nasa.gov/image/6870605/6870605~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0144/KSC-20190716-PH_FWM04_0144~orig.jpg
https://images-assets.nasa.gov/image/0101140/0101140~orig.jpg
https://images-assets.nasa.gov/image/KSC-20170127-PH_KLS02_0005/KSC-20170127-PH_KLS02_0005~orig.jpg
https://images-assets.nasa.gov/image/s69-32614/s69-32614~orig.jpg
https://images-assets.nasa.gov/image/s69-34039/s69-34039~orig.jpg
https://images-assets.nasa.gov/image/S69-62224/S69-62224~orig.jpg
https://images-assets.nasa.gov/image/S72-50438/S72-50438~orig.jpg
https://images-assets.nasa.gov/image/9309295/9309295~orig.jpg
https://images-assets.nasa.gov/image/as14-67-09364/as14-67-09364~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0085/KSC-20190716-PH_FWM04_0085~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0204/KSC-20190716-PH_FWM04_0204~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0393/KSC-20190716-PH_FWM04_0393~orig.jpg
https://images-assets.nasa.gov/image/S69-18569/S69-18569~orig.jpg
https://images-assets.nasa.gov/image/S66-36742/S66-36742~orig.jpg
https://images-assets.nasa.gov/image/jsc2007e045377/jsc2007e045377~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0088/KSC-20190716-PH_FWM04_0088~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0053/KSC-20190716-PH_FWM04_0053~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0018/KSC-20190716-PH_FWM04_0018~orig.jpg
https://images-assets.nasa.gov/image/KSC-20190716-PH_FWM04_0316/KSC-20190716-PH_FWM04_0316~orig.jpg
>>>

我在这里所做的有些天真，因为要获得原始大小的图像，我要做的就是清除每个缩略图URL末尾的~thumb.jpg，并将其替换为~orig.jpg -这假定原始文件具有JPEG扩展名。更好的解决方案是查看image["href"]，它是另一个JSON文件的URL，其中包含给定图像的所有不同版本/尺寸的URL。您只需向该JSON文件发出另一个请求，然后向该JSON中包含的一个URL发出请求。

答案 1 :(得分：1)

我们在images.nasa.gov页脚中有API文档：

https://images.nasa.gov/docs/images.nasa.gov_api_docs.pdf

检查一下，您应该能够使用我们在Web应用程序中所做的所有事情：我们的前端只是与API通讯的Angular －您可以对请求执行相同的操作。只需连接到

https://images-api.nasa.gov/

代替Web前端。

文档中包含使用Curl的示例，您应该可以将其转换为请求。我们自己使用请求来编写实用程序来操纵API。

玩得开心！

相关问题
最新问题