title: web crawler in rails,how to crawl all pages of the site
body: <p>I need to get  all urls from all pages of the given domain,<br/>
  I think it make sense to use background jobs, placing them on multiple queues<br/> 
  trying to use <a href="https://github.com/stewartmckee/cobweb" rel="nofollow">cobweb</a> but it seems very confusing gem, <br/>
  and <a href="http://anemone.rubyforge.org/" rel="nofollow">anomone</a>, anemone is working for a long time if there are a lot of pages<br/></p>
  
  <pre><code>require 'anemone'
  
  Anemone.crawl("http://www.example.com/") do |anemone|
    anemone.on_every_page do |page|
        puts page.links
    end
  end
  </code></pre>
  
  <p>What do u think would fit me best?</p>