title: Multiprocess : Persistent Pool?
body: <p>I have code like the one below :</p>
  <pre><code>def expensive(self,c,v):
      .....
  
  def inner_loop(self,c,collector):
      self.db.query('SELECT ...',(c,))
      for v in self.db.cursor.fetchall() :
          collector.append( self.expensive(c,v) ) 
  
  def method(self):
  
      # create a Pool
      #join the Pool ??
  
      self.db.query('SELECT ...')
      for c in self.db.cursor.fetchall() :
          collector = []
  
          #RUN the whole cycle in parallel in separate processes
          self.inner_loop(c, collector)
  
          #do stuff with the collector
  
      #! close the pool ?
  </code></pre>
  <p>both the Outer and the Inner loop are thousands of steps ...
  I think I understand how to run a Pool of couple of processes.
  All the examples I found show that more or less.</p>
  <p>But in my case I need to lunch a persistent Pool and then feed the data (c-value). Once a inner-loop process has finished I have to supply the next-available-c-value.
  And keep the processes running and collect the results.</p>
  <p>How do I do that ?</p>
  <hr />
  <p>A clunky idea I have is :</p>
  <pre><code>def method(self):
   ws = 4
   with Pool(processes=ws) as pool :
       cs = []
       for i,c in enumerate(..) :
         cs.append(c)  
         if i % ws == 0 :
           res = [pool.apply(self.inner_loop, (c)) for i in range(ws)]
           cs = []
           collector.append(res)
  </code></pre>
  <p>will this keep the same pool running !! i.e. not lunch new process every time ?i</p>
  <hr />
  <p>Do I need  'if i % ws == 0' part or I can use imap(), map_async() and the Pool obj will block the loop when available workers are exhausted and continue when some are freed ?</p>