title: Child process hanging multiprocessing
body: <p>I am having a problem where child processes are hanging in my python application, only 4/16 processes have finished all of these processes are adding to a multiprocessing queue. <a href="https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues" rel="nofollow">https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues</a> According to python docs:</p>
  
  <blockquote>
    <p>Warning</p>
    
    <p>As mentioned above, if a child process has put items on a queue (and
    it has not used JoinableQueue.cancel_join_thread), then that process
    will not terminate until all buffered items have been flushed to the
    pipe.</p>
    
    <p>This means that if you try joining that process you may get a deadlock
    unless you are sure that all items which have been put on the queue
    have been consumed. Similarly, if the child process is non-daemonic
    then the parent process may hang on exit when it tries to join all its
    non-daemonic children.</p>
    
    <p>Note that a queue created using a manager does not have this issue.
    See Programming guidelines.</p>
  </blockquote>
  
  <p>I believe this may be my problem, however I do a get() off the queue before I join.  I am not sure what other alternatives I can take.</p>
  
  <pre><code>def RunInThread(dictionary):
      startedProcesses = list()
      resultList = list()
      output = Queue()
      scriptList = ThreadChunk(dictionary, 16) # last number determines how many threads
  
      for item in scriptList:
          if __name__ == '__main__':
              proc = Process(target=CreateScript, args=(item, output))
              startedProcesses.append(proc)
              proc.start()
  
      while not output.empty():
          resultList.append(output.get())
  
      #we must wait for the processes to finish before continuing
      for process in startedProcesses:
          process.join()
          print "finished"
  
  #defines chunk of data each thread will process
  def ThreadChunk(seq, num):
    avg = len(seq) / float(num)
    out = []
    last = 0.0
  
    while last &lt; len(seq):
      out.append(seq[int(last):int(last + avg)])
      last += avg
  
    return out
  
  def CreateScript(scriptsToGenerate, queue):
      start = time.clock()
      for script in scriptsToGenerate:
      ...
          queue.put([script['timeInterval'], script['script']])
  
      print time.clock() - start
      print "I have finished"
  </code></pre>