Question

下面列出的是我写的刮刀的代码。我需要帮助为这个刮板添加延迟。我想每小时刮一页。

require 'open-uri'
require 'nokogiri'
require 'sanitize'

class Scraper

    def initialize(url_to_scrape)
        @url = url_to_scrape
    end

    def scrape
        # TO DO: change to JSON
        # page = Nokogiri::HTML(open(@url)) 
        puts "Initiating scrape..."
        raw_response = open(@url)
        json_response = JSON.parse(raw_response.read)
        page = Nokogiri::HTML(json_response["html"]) 

        # your page should now be a hash. You need the page["html"]

        # Change this to parse the a tags with the class "article_title"
        # and build the links array for each href in these article_title links
        puts "Scraping links..."
        links = page.css(".article_title")
        articles = []

        # everything else here should work fine.
        # Limit the number of links to scrape for testing phase
        puts "Building articles collection..."
        links.each do |link|
            article_url = "http://seekingalpha.com" + link["href"]
            article_page = Nokogiri::HTML(open(article_url))
            article = {}
            article[:company] = article_page.css("#about_primary_stocks").css("a")
            article[:content] = article_page.css("#article_content")
            article[:content] = Sanitize.clean(article[:content].to_s)
            unless article[:content].blank?
                articles << article
            end
        end

        puts "Clearing all existing transcripts..."
        Transcript.destroy_all
        # Iterate over the articles collection and save each record into the database
        puts "Saving new transcripts..."
        articles.each do |article|
            transcript = Transcript.new
            transcript.stock_symbol = article[:company].text.to_s
            transcript.content = article[:content].to_s
            transcript.save
        end

        #return articles
    end

end

Answer 1

那么当你完成抓取时你在做什么文章数组？

我不确定它是否是你要找的，但我只是使用cron来安排每小时运行一次这个脚本。如果你的脚本是一个更大的应用程序的一部分 - 有一个叫做whenever的整洁的宝石，它为cron任务提供了一个ruby包装。

希望有所帮助

如何用nokogiri每小时刮一个网站

1 个答案: