我正在使用BeautifulSoup和urlopen进行webscrape。当我尝试使用urlopen打开url时,提示出现大错误。我在做什么错了?
我使用python 3.7.2,我已经安装了Bs4和urlopen。
import bs4
from urllib.request import urlopen
from bs4 import BeautifulSoup as soup
my_url = 'https://ebgames.com.au/search?q=Skyrim'
var = urlopen(my_url)
ERROR:
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
urlopen (my_url)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
预期结果没有错误。
答案 0 :(得分:0)
尝试将浏览器用户代理设置为您的请求作为请求标头:
import bs4
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as soup
my_url = req = Request('https://ebgames.com.au/search?q=Skyrim', headers={'User-Agent': 'Mozilla/5.0'})
var = urlopen(my_url).read()
有关更多信息,请参阅此answer