title: Finding the cause of a BrokenProcessPool in python's concurrent.futures
body: <p><strong>In a nutshell</strong></p>
  
  <p>I get a <code>BrokenProcessPool</code> exception when parallelizing my code with <code>concurrent.futures</code>. No further error is displayed. I want to find the cause of the error and ask for ideas of how to do that. </p>
  
  <p><strong>Full problem</strong></p>
  
  <p>I am using <a href="https://docs.python.org/3.5/library/concurrent.futures.html" rel="noreferrer">concurrent.futures</a> to parallelize some code. </p>
  
  <pre><code>with ProcessPoolExecutor() as pool:
      mapObj = pool.map(myMethod, args)
  </code></pre>
  
  <p>I end up with (and only with) the following exception:</p>
  
  <pre><code>concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore
  </code></pre>
  
  <p>Unfortunately, the program is complex and the error appears only after the program has run for 30 minutes. Therefore, I cannot provide a nice minimal example. </p>
  
  <p>In order to find the cause of the issue, I wrapped the method that I run in parallel with a try-except-block:</p>
  
  <pre><code>def myMethod(*args):
      try:
          ...
      except Exception as e:
          print(e)
  </code></pre>
  
  <p>The problem remained the same and the except block was never entered. I conclude that the exception does not come from my code. </p>
  
  <p>My next step was to write a custom <code>ProcessPoolExecutor</code> class that is a child of the original <code>ProcessPoolExecutor</code> and allows me to replace some methods with cusomized ones. I copied and pasted the original code of the method <code>_process_worker</code> and added some print statements. </p>
  
  <pre><code>def _process_worker(call_queue, result_queue):
      """Evaluates calls from call_queue and places the results in result_queue.
          ...
      """
      while True:
          call_item = call_queue.get(block=True)
          if call_item is None:
              # Wake up queue management thread
              result_queue.put(os.getpid())
              return
          try:
              r = call_item.fn(*call_item.args, **call_item.kwargs)
          except BaseException as e:
                  print("??? Exception ???")                 # newly added
                  print(e)                                   # newly added
              exc = _ExceptionWithTraceback(e, e.__traceback__)
              result_queue.put(_ResultItem(call_item.work_id, exception=exc))
          else:
              result_queue.put(_ResultItem(call_item.work_id,
                                           result=r))
  </code></pre>
  
  <p>Again, the <code>except</code> block is never entered. This was to be expected, because I already ensured that my code does not raise an exception (and if everything worked well, the exception should be passed to the main process).</p>
  
  <p>Now I am lacking ideas how I could find the error. The exception is raised here:</p>
  
  <pre><code>def submit(self, fn, *args, **kwargs):
      with self._shutdown_lock:
          if self._broken:
              raise BrokenProcessPool('A child process terminated '
                  'abruptly, the process pool is not usable anymore')
          if self._shutdown_thread:
              raise RuntimeError('cannot schedule new futures after shutdown')
  
          f = _base.Future()
          w = _WorkItem(f, fn, args, kwargs)
  
          self._pending_work_items[self._queue_count] = w
          self._work_ids.put(self._queue_count)
          self._queue_count += 1
          # Wake up queue management thread
          self._result_queue.put(None)
  
          self._start_queue_management_thread()
          return f
  </code></pre>
  
  <p>The process pool is set to be broken here:</p>
  
  <pre><code>def _queue_management_worker(executor_reference,
                               processes,
                               pending_work_items,
                               work_ids_queue,
                               call_queue,
                               result_queue):
      """Manages the communication between this process and the worker processes.
          ...
      """
      executor = None
  
      def shutting_down():
          return _shutdown or executor is None or executor._shutdown_thread
  
      def shutdown_worker():
          ...
  
      reader = result_queue._reader
  
      while True:
          _add_call_item_to_queue(pending_work_items,
                                  work_ids_queue,
                                  call_queue)
  
          sentinels = [p.sentinel for p in processes.values()]
          assert sentinels
          ready = wait([reader] + sentinels)
          if reader in ready:
              result_item = reader.recv()
          else:                               #THIS BLOCK IS ENTERED WHEN THE ERROR OCCURS
              # Mark the process pool broken so that submits fail right now.
              executor = executor_reference()
              if executor is not None:
                  executor._broken = True
                  executor._shutdown_thread = True
                  executor = None
              # All futures in flight must be marked failed
              for work_id, work_item in pending_work_items.items():
                  work_item.future.set_exception(
                      BrokenProcessPool(
                          "A process in the process pool was "
                          "terminated abruptly while the future was "
                          "running or pending."
                      ))
                  # Delete references to object. See issue16284
                  del work_item
              pending_work_items.clear()
              # Terminate remaining workers forcibly: the queues or their
              # locks may be in a dirty state and block forever.
              for p in processes.values():
                  p.terminate()
              shutdown_worker()
              return
          ...
  </code></pre>
  
  <p>It is (or seems to be) a fact that a process terminates, but I have no clue why. Are my thoughts correct so far? <strong>What are possible causes that make a process terminate without a message? (Is this even possible?) Where could I apply further diagnostics? Which questions should I ask myself in order to come closer to a solution?</strong> </p>
  
  <p>I am using python 3.5 on 64bit Linux. </p>