title: Python multiprocessing with an updating queue and an output queue
body: <p>How can I script a Python multiprocess that uses two Queues as these ones?:</p>
  
  <ol>
  <li>one as a working queue that starts with some data and that, depending on conditions of the functions to be parallelized, receives further tasks on the fly, </li>
  <li>another that gathers results and is used to write down the result after processing finishes.</li>
  </ol>
  
  <p>I basically need to put some more tasks in the working queue depending on what I found in its initial items. The example I post below is silly (I could transform the item as I like and put it directly in the output Queue), but its mechanics are clear and reflect part of the concept I need to develop.</p>
  
  <p>Hereby my attempt:</p>
  
  <pre><code>import multiprocessing as mp
  
  def worker(working_queue, output_queue):
      item = working_queue.get() #I take an item from the working queue
      if item % 2 == 0:
          output_queue.put(item**2) # If I like it, I do something with it and conserve the result.
      else:
          working_queue.put(item+1) # If there is something missing, I do something with it and leave the result in the working queue 
  
  if __name__ == '__main__':
      static_input = range(100)    
      working_q = mp.Queue()
      output_q = mp.Queue()
      for i in static_input:
          working_q.put(i)
      processes = [mp.Process(target=worker,args=(working_q, output_q)) for i in range(mp.cpu_count())] #I am running as many processes as CPU my machine has (is this wise?).
      for proc in processes:
          proc.start()
      for proc in processes:
          proc.join()
      for result in iter(output_q.get, None):
          print result #alternatively, I would like to (c)pickle.dump this, but I am not sure if it is possible.
  </code></pre>
  
  <p>This does not end nor print any result.</p>
  
  <p>At the end of the whole process I would like to ensure that the working queue is empty, and that all the parallel functions have finished writing to the output queue before the later is iterated to take out the results. Do you have suggestions on how to make it work?</p>