title: Python: Multiprocessing Recipe for Queue(s) with Many Consumers
body: <p>Since I have been wasting a lot of time trying to <code>join()</code> workers in python <code>multiprocessing</code> architectures that get their tasks from a <code>multiprocessing.Queue</code>, which is at the same time fed by a feeder function through <code>put()</code>:</p>
  <p><strong>Can someone contribute a short but robust recipe for this kind of architecture?</strong></p>
  <p>Let's say up to 10 feeders and up to 100 workers. Queue items might be large.</p>
  <p>I can only guess that in the past I have often had the queue busy and not responding anymore, but mostly, I had the task done, only the jobs never joined, or wether they joined or not seemed to depend on arbitrary parameters.</p>
  <p>So imagine the following workflow:</p>
  <ul>
  <li>One or more <code>feeder()</code> jobs read from input (i.e. disk) and create tasks for the first line of workers. They put the tasks into a queue <code>Q</code>. When there are no more tasks to create, the feeders should join.</li>
  <li>Many workers take the tasks out of <code>Q</code> and process it. Then they put the result into another queue <code>R</code>. When these workers fail to get more jobs from <code>Q</code> and are done with the last task, they should join. Consider that they might see an exception due to invalid input/task. They should still join.</li>
  <li>Let's add another line of workers that takes always two results from <code>R</code>, merges them and puts the result back into <code>R</code>. When there is only one result left in <code>R</code>, all workers of the second line should join.</li>
  <li>Finally I can take the last remaining aggregated result out of <code>R</code> and be happy.</li>
  </ul>
  <p>I think this setup should include enough aspects to allow for generalization of many different tasks.</p>
  <p>To avoid blocking of the queue(s), I have already tried out the following ideas:</p>
  <ul>
  <li>Use multiple queues with always putting to the smallest queue and getting from <strong>random</strong> queue</li>
  <li>Do not block in <code>get()</code>, but try getting without wait, if fails, sleep for a short time and retry, keep track of number of tries, after a while, give up, assuming queue is empty.</li>
  </ul>
  <p>I can add some code, but maybe this would lead to bug fixing instead of sharing best practices.</p>