title: python multiprocessing .join() deadlock depends on worker function
body: <p>I am using the <a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow noreferrer"><code>multiprocessing</code></a> python library to spawn 4 <a href="https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process" rel="nofollow noreferrer"><code>Process()</code></a> objects to parallelize a cpu intensive task. The task (inspiration and code from this great <a href="http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/" rel="nofollow noreferrer">article</a>) is to compute the prime factors for every integer in a list.</p>
  
  <p><strong>main.py:</strong></p>
  
  <pre><code>import random
  import multiprocessing
  import sys
  
  num_inputs  = 4000
  num_procs   = 4
  proc_inputs = num_inputs/num_procs
  input_list  = [int(1000*random.random()) for i in xrange(num_inputs)]
  
  output_queue = multiprocessing.Queue()
  procs        = []
  for p_i in xrange(num_procs):
    print "Process [%d]"%p_i
    proc_list = input_list[proc_inputs * p_i:proc_inputs * (p_i + 1)]
    print " - num inputs: [%d]"%len(proc_list)
  
    # Using target=worker1 HANGS on join
    p = multiprocessing.Process(target=worker1, args=(p_i, proc_list, output_queue))
    # Using target=worker2 RETURNS with success
    #p = multiprocessing.Process(target=worker2, args=(p_i, proc_list, output_queue))
  
    procs.append(p)
    p.start()
  
  for p in jobs:
    print "joining ", p, output_queue.qsize(), output_queue.full()
    p.join()
    print "joined  ", p, output_queue.qsize(), output_queue.full()
  
  print "Processing complete."
  ret_vals = []
  while output_queue.empty() == False:
      ret_vals.append(output_queue.get())
  print len(ret_vals)
  print sys.getsizeof(ret_vals)
  </code></pre>
  
  <p><strong>Observation:</strong></p>
  
  <ul>
  <li>If the target for each process is the function <code>worker1</code>, for an input list larger than 4000 elements the main thread gets stuck on <code>.join()</code>, waiting for the spawned processes to terminate and never returns.</li>
  <li>If the target for each process is the function <code>worker2</code>, for the same input list the code works just fine and the main thread returns.</li>
  </ul>
  
  <p>This is very confusing to me, as the only difference between <code>worker1</code> and <code>worker2</code> (see below) is that the former inserts individual lists in the <code>Queue</code> whereas the latter inserts a single list of lists for each process.</p>
  
  <p>Why is there deadlock using <code>worker1</code> and not using <code>worker2</code> target?
  Shouldn't both (or neither) go beyond the <a href="https://stackoverflow.com/questions/5900985/multiprocessing-queue-maxsize-limit-is-32767">Multiprocessing Queue maxsize limit is 32767</a>?</p>
  
  <hr>
  
  <p><strong>worker1 vs worker2:</strong></p>
  
  <pre><code>def worker1(proc_num, proc_list, output_queue):
      '''worker function which deadlocks'''  
      for num in proc_list:
          output_queue.put(factorize_naive(num))
  
  def worker2(proc_num, proc_list, output_queue):
      '''worker function that works'''
      workers_stuff = []
  
      for num in proc_list:
          workers_stuff.append(factorize_naive(num))
      output_queue.put(workers_stuff)
  </code></pre>
  
  <hr>
  
  <p>There are a <strong>lot</strong> of similar questions on SO, but I believe the core of this questions is clearly distinct from all of them.</p>
  
  <p><strong>Related Links:</strong></p>
  
  <ul>
  <li><a href="https://sopython.com/canon/82/programs-using-multiprocessing-hang-deadlock-and-never-complete/" rel="nofollow noreferrer">https://sopython.com/canon/82/programs-using-multiprocessing-hang-deadlock-and-never-complete/</a></li>
  <li><a href="https://stackoverflow.com/questions/29681261/python-multiprocessing-issues">python multiprocessing issues</a></li>
  <li><a href="https://stackoverflow.com/questions/21641887/python-multiprocessing-process-hangs-on-join-for-large-queue">python multiprocessing - process hangs on join for large queue</a></li>
  <li><a href="https://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers">Process.join() and queue don&#39;t work with large numbers</a></li>
  <li><a href="https://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em">Python 3 Multiprocessing queue deadlock when calling join before the queue is empty</a></li>
  <li><a href="https://stackoverflow.com/questions/26738648/script-using-multiprocessing-module-does-not-terminate">Script using multiprocessing module does not terminate</a></li>
  <li><a href="https://stackoverflow.com/questions/33152171/why-does-multiprocessing-process-join-hang">Why does multiprocessing.Process.join() hang?</a></li>
  <li><a href="https://stackoverflow.com/questions/14429703/when-to-call-join-on-a-process">When to call .join() on a process?</a></li>
  <li><a href="https://stackoverflow.com/questions/25391025/what-exactly-is-python-multiprocessing-modules-join-method-doing">What exactly is Python multiprocessing Module&#39;s .join() Method Doing?</a></li>
  </ul>