title: Python wait for processes in multiprocessing Pool to complete without either closing Pool or use map()
body: <p>I have a code piece like below</p>
  
  <pre><code>pool = multiprocessing.Pool(10)
  for i in range(300):
      for m in range(500):
          data = do_some_calculation(resource)
          pool.apply_async(paralized_func, data, call_back=update_resource)
      # need to wait for all processes finish
      # {...}
      # Summarize resource
      do_something_with_resource(resource)
  </code></pre>
  
  <p>So basically I have 2 loops. I init process pool once outside the loops to avoid overheating. At the end of 2nd loop, I want to summarize the result of all processes.</p>
  
  <p>Problem is that I can't use <code>pool.map()</code> to wait because of variation of <code>data</code> input. I can't use <code>pool.join()</code> and <code>pool.close()</code> either because I still need to use the <code>pool</code> in next iteration of 1st loop.</p>
  
  <p>What is the good way to wait for processes to finish in this case?</p>
  
  <p>I tried checking for pool._cache at the end of 2nd loop.</p>
  
  <pre><code>while len(process_pool._cache) &gt; 0:
      sleep(0.001)
  </code></pre>
  
  <p>This way works but look weird. Is there a better way to do this?</p>