Question

您将看到一系列问题，我建立了一个小机械化任务来访问page（），找到咖啡馆的链接，并将咖啡馆的详细信息保存在csv中。

 task :estimateone => :environment do
  require 'mechanize'
  require 'csv'

  mechanize = Mechanize.new
  mechanize.history_added = Proc.new { sleep 30.0 }
  mechanize.ignore_bad_chunking = true
  mechanize.follow_meta_refresh = true
  page = mechanize.get('http://www.siteexamplea.com/city/list/50-city-cafes-you-should-have-eaten-breakfast-at')
  results = []
  results << ['name', 'streetAddress', 'addressLocality', 'postalCode', 'addressRegion', 'addressCountry', 'telephone', 'url']
  page.css('ol li a').each do |link|
   mechanize.click(link)

   name = mechanize.page.css('article h1[itemprop="name"]').text.strip
   streetAddress = mechanize.page.css('address span span[itemprop="streetAddress"]').text.strip
   addressLocality = mechanize.page.css('address span span[itemprop="addressLocality"]').text.strip
   postalCode = mechanize.page.css('address span span[itemprop="postalCode"]').text.strip
   addressRegion = mechanize.page.css('address span span[itemprop="addressRegion"]').text.strip
   addressCountry = mechanize.page.css('address span meta[itemprop="addressCountry"]').text.strip
   telephone = mechanize.page.css('address span[itemprop="telephone"]').text.strip
   url = mechanize.page.css('article p a[itemprop="url"]').text.strip
   tags = mechanize.page.css('article h1[itemprop="name"]').text.strip

    results << [name, streetAddress, addressLocality, postalCode, addressRegion, addressCountry, telephone, url]
  end

  CSV.open("filename.csv", "w+") do |csv_file|
    results.each do |row|
      csv_file << row
    end
  end
end

当我到达第十个链接时，我遇到了503错误。

Mechanize::ResponseCodeError: 503 => Net::HTTPServiceUnavailable for https://www.city.com/city/directory/morning-after -- unhandled response

我已经尝试了几种方法来阻止这种情况的发生或从这种状态中解救出来，但我无法解决。有提示吗？

Answer 1

您想在失败的请求just like here

下进行救援

task :estimateone => :environment do
  require 'mechanize'
  require 'csv'

  begin
  # ...
  page = mechanize.get('http://www.theurbanlist.com/brisbane/a-list/50-brisbane-cafes-you-should-have-eaten-breakfast-at')
  rescue Mechanize::ResponseCodeError
    # do something with the result, log it, write it, mark it as failed, wait a bit and then continue the job
    next
  end
end

我的猜测是您正在达到API速率限制。这不会解决您的问题，因为它不在您身边，而是在服务器端。但可以提供更多的工作范围，因为您现在可以标记无效的链接并从那里继续。

机械化处理错误

1 个答案: