Question

我试图制作一个脚本，从我最喜欢的图像板中的线程下载所有图像或视频：2ch.hk
我成功了，直到我想异步下载这些文件（例如，为了提高性能）这是代码http://ideone.com/k2l4Hm

file = http.get(source).body
require 'net/http'
multithreading = false
Net::HTTP.start("2ch.hk", :use_ssl => true) do |http|
 thread = http.get("/b/res/133467978.html").body
 sources = []
 thread.scan(/<a class="desktop" target="_blank" href=".+">.+<\/a>/).each do |a|
    source = "/b#{/<a class="desktop" target="_blank" href="\.\.(.+)">.+<\/a>/.match(a).to_a[1]}"
    sources << source
  end
  i = 0
  start = Time.now
  if multithreading
    threads = []
    sources.each do |source|
      threads << Thread.new(i) do |j|
        file = http.get(source).body #breaks everything
        # type = /.+\.(.+)/.match(source)[1]
        # open("#{j}.#{type}","wb") { |new_file|
        #   new_file.write(file)
        # }
      end
      i += 1
    end
    threads.each do |thr|
      thr.join
    end
    # until downloade=sources.size
    #
    # end
  else
    sources.each do |source|
      file = http.get(source).body
      type = /.+\.(.+)/.match(source)[1]
      open("#{i}.#{type}","wb") { |new_file|
        new_file.write(file)
      }
      i += 1
      print "#{(((i).to_f / sources.size) * 100).round(2)}% "
    end
    puts
  end
  puts "Done. #{i} files were downloaded. It took #{Time.now - start} seconds"
end

我想这条线会崩溃一切。

file = http.get(source).body

或许这就是问题。

threads.each do |thr|
  thr.join
end

错误消息总是不同，从错误的文件描述符和IO错误到“您可能在Ruby解释器或扩展库中遇到了错误。”
如果你想尝试运行我的代码，请用新线程（来自2ch.hk/b）替换链接到第4行的线程，因为我的代码中的那个可能会在你运行我的代码时被删除
ruby版本：2.3.1，OS Xubuntu 16.10

Answer 1

使用支持并行请求的ruby http lib可能会有更好的性能：

https://github.com/typhoeus/typhoeus

e.g。

ionicBootstrap(MyApp, [], {      
    tabsHideOnSubPages:false
});

Answer 2

我的代码存在的问题是我无法同时在Net :: HTTP实例上发出多个请求。解决方案是为每个线程打开HTTP连接。

异步下载文件

2 个答案: