title: Web crawler in Ruby: How to achieve the best perfomance?
body: <p>I'm writing a web-crawler that should be able to parse multiple pages at the same time. I use Nokogiri for parsing which is quiet good and solve all my tasks, but I don't know how to achieve better perfomance. </p>
  
  <p>I use threads to make many open-uri requests at the same time and it makes the process quicker, but it seems that it's still far from the potential that I can achieve from a single server. Should I use multiple processes? What are the limits of the threads and processes that can be launched for a single ruby application? </p>
  
  <p>By the other words: how to achieve the best performance in this case.</p>