Webscrape;循环遍历多个网址

时间：2018-02-17 18:17:11

标签： python loops beautifulsoup

我试图从TripAdvisor获取评分和评论。我能够从第一页获得评分和评论。 

# find the whole block of code that includes the review and the rating_bubble
reviews = soup.find_all('div', 'reviewSelector')


# parse the ratings and reviews from the block of code for first page
for r in reviews:
    print("Rating: ", int(r.find('span','ui_bubble_rating')['class']
    [1].split('_')[1])/10)
    print("Review snipet: ", r.p.text)

我还能够获取每个连续页面的URL：

for i in range(0, 200, 5):
    url = 'https://www.tripadvisor.com/Hotel_Review-g42139-d2156247-Reviews-
    or'+str(i)+'-Roberts_Riverwalk_Hotel_Detroit-Detroit_Michigan.html'
    urls = print(url)

现在我想知道循环显示给定网址以获取每个网页的评分和评论的最佳方式。

有人有任何建议吗？

2 个答案:

答案 0 :(得分：0)

通常我只是首先下载页面，然后通过标记进行解析，如果它甚至可以在不使用Selenium或phantomjs之类的情况下进行解析。这是一个想法：

import requests


headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

for i in range(0, 200, 5):
    url = 'https://www.tripadvisor.com/Hotel_Review-g42139-d2156247-Reviews-or{}-Roberts_Riverwalk_Hotel-Detroit_Michigan.html'.format(i)
    response = requests.get(url, headers=headers )
    with open('ta_reviews_{}.{}'.format(i, 'html'), 'w') as f:
        f.write(response.text)

答案 1 :(得分：0)

为什么不使用您在for循环中为一个页面编写的代码？

这样的事情会：

for i in range(0, 200, 5):
    r = requests.get('https://www.tripadvisor.in/Hotel_Review-g42139-d2156247-Reviews-or{}-Roberts_Riverwalk_Hotel-Detroit_Michigan.html'.format(i))
    soup = BeautifulSoup(r.text, 'lxml')
    for r in soup.find_all('div', 'reviewSelector'):
        print("Rating: ", int(r.find('span', 'ui_bubble_rating')['class'][1].split('_')[1])/10)
        print("Review snipet: ", r.p.text)

部分输出：

Rating:  1.0
Review snipet:  These are the only GOOD things I can say: 1) the bellhop was kind 2) the view of the water was nice 3) the shower was hot. 
BUT.....we had to use the hot shower to heat the room because there was NO HEAT!! In Detroit....In...More
Rating:  2.0
Review snipet:  This appears to have once been a grand old hotel. It is still beautiful but needs a lot of rehab. It took us over 2 hours to get our room. The room was quite reasonably clean however, a curtain was fallen down and a lampshade...More
Rating:  4.0
Review snipet:  Spacious hotel with large rooms, grand lobby and excellent service.  Everything that
was needed in the room was there.  Can not understand multiple unfavorable reviews.
We loved our stay. Room view of river and walk were a highlight of our stay.
Rating:  3.0
Review snipet:  Had my family reunion in August, 2017. Most of the rooms had no air condition. Staff was friendly, but took at least 20 min to check-in. Rooms was not clean on a timely basis. Overall service was awful.
Rating:  1.0
Review snipet:  The highlight of the room here was the two-foot stain in the middle of the carpet. Gross. The room was not the river-view we had reserved, and when asked for such a room, were told it was not available. This perhaps once nice hotel needs...More
Rating:  1.0
Review snipet:  Very disappointed. I had a reservation booked weeks ahead of time only to have my reservation cancelled by the hotel only 3 hours before check-in. I will never book a stay here again. There are many, many better hotel options in downtown Detroit.
Rating:  1.0
Review snipet:  Booked a river view room online. Took 20 minutes to get checked in. Got to the room and there wasn’t a river view, instead a brick wall out of my window and the room was freezing. Thermostat said 62 deg. Went to front desk and...More
Rating:  1.0
Review snipet:  The riverview room I reserved in advance for 2 nights (and paid for) was not available at check-in, so the clerk gave me a down-graded room, without apology or adjustment in rate.  It was shabby and musty, dirty carpet, fixtures coming loose in the bathroom,...More
Rating:  1.0
Review snipet:  Everything about our stay here was terrible. We had booked a room with two queen beds, as there were 3 of us traveling. It was raining while when we arrived, and the ceiling was leaking over one of the beds in our room, as well...More
Rating:  1.0
Review snipet:  Awful, would be an understatement of what we went through our stay. Once we arrived at the hotel from our drive from cleveland, I was excited because of how great this location was. It was on the river, close to downtown and within our budget....More
Rating:  4.0
Review snipet:  We enjoyed our stay, staff were pleasant and helpful in suggesting restaurants for dinner. Our room was very spacious with high ceilings. The location is great. I suggest you request a room overlooking the waterfront. Nearby walks were very pleasant. We are planning another visit...More
Rating:  5.0
Review snipet:  I know this place has some bad reviews, but let me tell you! We were there in game day and it was busy, the women that checked us in was amazing. We were a little early and she was nice enough to see if our...More
Rating:  1.0
Review snipet:  My husband and I booked a three night stay here. I would give this hotel a ZERO, but I had to give at least one star to give a review. The website pictures are deceiving as far as the inside of the hotel. Once you...More
Rating:  1.0
Review snipet:  We try to find quaint, boutique hotels on, or near the water when we travel extensively. Never have I been more disappointed than Robert Riverwalk Hotel in Detroit.
A beautiful waterfront recently updated and from the outside, this former pharmaceutical office, transformed to hotel looked...More
Rating:  1.0
Review snipet:  What might have been a quirky, off-the-beaten path place to stay ultimately proved to be the worst hotel experience I've ever had. The moment we arrived, my friend and I were given the stink-eye for asking to check in early. We were never welcomed nor...More
Rating:  1.0
Review snipet:  If I could give this hotel a Zero I would! We stepped into the hotel & we were not greeted & we asked the lady at the front desk if we could get a rolling cart for our bags, she told us we had to...More
Rating:  1.0
Review snipet:  Check in staff was very rude when we finally got in our room there was no refridge or micro wave the bed didn't even have a comforter on it the TV only had 6 channels. With in 15 minutes another couple entered our room claiming...More
Rating:  1.0
Review snipet:  The list of problems that we had with this hotel filled up a whole page!
I have submitted a "Request For Refund" which I changed to "Demand for Refund".
It took 40 minutes to check in due to an IT problem.
First we were given...More
Rating:  1.0
Review snipet:  We reserved a queen room. Called to double check it was okay for today. Arrived and said that they over booked and had no rooms. We're not helpful with a solution on what we were suppose to do as we prepaid.
Rating:  2.0
Review snipet:  We ended up getting this hotel through a blind hotwire deal.  I was worried after reading the reviews but kept an open mind.  There is a lot of work being done in the neighbourhood surrounding the hotel.  Nice condo's and beautiful brownstone townhouses etc.  I...More
相关问题
最新问题