English 中文(简体)
Threading in Ruby with a limit
原标题:

I have a task I need to perform, do_stuff(opts), that will take ~1s each, even while 1 - 10 of them are running in parallel. I need to collect an array of the results for each operation at the end.

If I have 30 stuffs to do, how would I use threading effectively to queue up the do_stuff(opts) operations so no more than 10 are running concurrently, but the array of results is not given/printed/etc until all (30) tasks have been completed?

I usually have at least some code to try and illustrate what I mean, but with threading I m at a bit of a loss! Thanks in advance

最佳回答

You need to implement this pattern
This question discusses how that can be done in Ruby

问题回答

I dunno how well it ll work for a more complex application, but I found something like this to work nicely for a simple threading scenario with macruby.

thread_limit = 4

threads = []
things_to_process.each do |thing|
  until threads.map { |t| t.status }.count("run") < thread_limit do sleep 5 end
  threads << Thread.new { the_task(thing) }
end
output = threads.map { |t| t.value }

the until loop waits around until there are less than the specified number of created threads running before allowing execution of the main thread to continue on to start the next thread.

the output variable will be assigned an array of the values returned by the_task with an ordering corresponding to the input array things_to_process. The main thread will block until every created thread returns a value.

If you re really after performance, you might also want to look into jruby.
It uses actual OS threads and not the green threads the other ruby implementations use

This solution gathers results in $results array. It allows thread_limit threads to be created, then waits on them to complete before creating any more.

   $results = []

   def do_stuff(opts={})
      done 
   end

   def thread_wait(threads)
     threads.each{|t| t.join}
     threads.each {|t| $results << t }
     threads.delete_if {|t| t.status == false}
     threads.delete_if {|t| t.status.nil? }
   end

   opts = {}
   thread_limit = 20
   threads = []
   records.each do |r|
     thread_wait(threads) while threads.length >= thread_limit
     t = Thread.new { do_stuff(opts) }
     t.abort_on_exception = true
     threads << t
   end
   # Ensure remaining threads complete
   threads.each{|t| t.join}

Also, take a look at this tutorial if you are new to Ruby threads.

I use parals and paralsmap:

def parals(objects, n: 50)
  objects.shuffle.each_slice(n).map do |g|
    print  { 
    threads = []
    g.map { |i| threads << Thread.new(i) { |i| yield(i) } }
    threads.each(&:join)
    print  } 
  end
end

def paralsmap(objects, n: 50)
  res = []

  objects.each_slice(n).map do |g|
    print  { 
    threads = []
    g.map { |i| threads << Thread.new(i, res) { |i| res << yield(i) } }
    threads.each(&:join)
    print  } 
  end

  res
end

e.g.:

parals((0..100).to_a) { |i| puts i }
urls = parals((0..100).to_a) { |i| "https://google.com/?q=#{i}" }

You can use the n parameter to limit the number of threads.





相关问题
Ruby parser in Java

The project I m doing is written in Java and parsers source code files. (Java src up to now). Now I d like to enable parsing Ruby code as well. Therefore I am looking for a parser in Java that parses ...

rails collection_select vs. select

collection_select and select Rails helpers: Which one should I use? I can t see a difference in both ways. Both helpers take a collection and generates options tags inside a select tag. Is there a ...

RubyCAS-Client question: Rails

I ve installed RubyCAS-Client version 2.1.0 as a plugin within a rails app. It s working, but I d like to remove the ?ticket= in the url. Is this possible?

Ordering a hash to xml: Rails

I m building an xml document from a hash. The xml attributes need to be in order. How can this be accomplished? hash.to_xml

multiple ruby extension modules under one directory

Can sources for discrete ruby extension modules live in the same directory, controlled by the same extconf.rb script? Background: I ve a project with two extension modules, foo.so and bar.so which ...

Text Editor for Ruby-on-Rails

guys which text editor is good for Rubyonrails? i m using Windows and i was using E-Texteditor but its not free n its expired now can anyone plese tell me any free texteditor? n which one is best an ...

热门标签