title: Asynchronous multiprocessing with a worker pool in Python: how to keep going after timeout?
body: <p>I would like to run a number of jobs using a pool of processes and apply a given timeout after which a job should be killed and replaced by another working on the next task. </p>
  
  <p>I have tried to use the <code>multiprocessing</code> module which offers a method to run of pool of workers asynchronously (e.g. using <code>map_async</code>), but there I can only set a "global" timeout after which all processes would be killed. </p>
  
  <p>Is it possible to have an individual timeout after which <strong>only a single process that takes too long is killed and a new worker is added</strong> to the pool again instead (<strong>processing the next task and skipping the one that timed out</strong>)?</p>
  
  <p>Here's a simple example to illustrate my problem:</p>
  
  <pre><code>def Check(n):
    import time
    if n % 2 == 0: # select some (arbitrary) subset of processes
      print "%d timeout" % n
      while 1:
        # loop forever to simulate some process getting stuck
        pass
    print "%d done" % n
    return 0
  
  from multiprocessing import Pool
  pool = Pool(processes=4)
  result = pool.map_async(Check, range(10))
  print result.get(timeout=1)    
  </code></pre>
  
  <p>After the timeout all workers are killed and the program exits. I would like instead that it continues with the next subtask. Do I have to implement this behavior myself or are there existing solutions?</p>
  
  <h2>Update</h2>
  
  <p>It is possible to kill the hanging workers and they are automatically replaced. So I came up with this code:</p>
  
  <pre><code>jobs = pool.map_async(Check, range(10))
  while 1:
    try:
      print "Waiting for result"
      result = jobs.get(timeout=1)
      break # all clear
    except multiprocessing.TimeoutError: 
      # kill all processes
      for c in multiprocessing.active_children():
        c.terminate()
  print result
  </code></pre>
  
  <p>The problem now is that the loop never exits; even after all tasks have been processed, calling <code>get</code> yields a timeout exception. </p>