title: Rails - loop curls request eating memory
body: <p>I use the gem <code>Curb</code> (tried too with <code>httparty</code>) to performing a lot of http request and this working good. But in one of my (<code>rake</code>) task (where I do 20k+ requests) I have a memory problem (Rails "eats" more than 2GB of RAM until there is no free memory anymore).</p>
  
  <p>Seems that Rails "don't wait" for a response and go ahead in another thread in the loop, the problem is that in this manner there will be created a lot of objects not collected by the garbage collector (I think) and is the reason of the memory leak. </p>
  
  <p>There is a method to say to rails to wait until the response is came? (I tried with <code>sleep</code> but is not a stable solution).</p>
  
  <p>I have a pseudocode like this:</p>
  
  <pre><code>def the_start
    while start_date &lt;= end_date do                  # ~ 140 loop 
      a_method_that_do_sub_specifics_call
    end
  end
  
  def a_method_that_do_sub_specifics_call
      some_data.each do |r|                          # ~ 180 loop
          do_a_call
          #do something with models (update/create entries,...)
      end
  end
  
  def do_a_call                                      # called ~ 25k times
      # with the gem Curb version
      req = Curl::Easy.new do |curl| 
         curl.ssl_verify_peer = false
         curl.url = url
         curl.headers['Content-type'] = 'application/json'
      end
      req.perform
  
      # actual version, with httparty gem
      req = HTTParty.get("#{url}",
          :headers =&gt; {'Content-type' =&gt; 'application/json'})
  end
  </code></pre>
  
  <p>Seems that Rails doesn't wait to have the results of <code>req.perform</code>.</p>
  
  <p><strong>EDIT:</strong><br>
  Tried too to instanciate only once the <code>Curl::Easy</code> object, using <code>Curl::Easy.perform()</code> and <code>req.close</code> (that should calls implicitly GC) after the call but without success ever big memory usage. The only solution that (I think) can work is to "blocks" rails until the response is came, but how?</p>
  
  <p><strong>EDIT 2</strong><br>
  In another task I call only the <code>a_method_that_do_sub_specifics_call</code> without problems.</p>
  
  <p><strong>EDIT 3</strong><br>
  After some performance mod (placing <code>find_each(:batch_size =&gt; ...)</code>, <code>GC.start</code>,...) the task works a little better.. now the first ~100 loop (<code>do_a_call</code>) work good, after that the memory usage jump from 100Mb to 2Gb+ again.</p>