title: Periodically restart Python multiprocessing pool
body: <p>I have a Python multiprocessing pool doing a very long job that even after a thorough debugging is not robust enough not to fail every 24 hours or so, because it depends on many third-party, non-Python tools with complex interactions. Also, the underlying machine has certain problems that I cannot control. Note that by failing I don't mean the whole program crashing, but some or most of the processes becoming idle because of some errors, and the app itself either hanging or continuing the job just with the processes that haven't failed.</p>
  <p>My solution right now is to periodically kill the job, manually, and then just restart from where it was.</p>
  <p>Even if it's not ideal, what I want to do now is the following: restart the multiprocessing pool periodically, programatically, from the Python code itself. I don't really care if this implies killing the pool workers in the middle of their job. Which would be the best way to do that?</p>
  <p>My code looks like:</p>
  <pre><code>with Pool() as p:
      for _ in p.imap_unordered(function, data):
          save_checkpoint()
          log()
  </code></pre>
  <p>What I have in mind would be something like:</p>
  <pre><code>start = 0
  end = 1000  # magic number
  while start + 1 &lt; len(data):
      current_data = data[start:end]
      with Pool() as p:
          for _ in p.imap_unordered(function, current_data):
              save_checkpoint()
              log()
              start += 1
              end += 1
  </code></pre>
  <p>Or:</p>
  <pre><code>start = 0
  end = 1000  # magic number
  while start + 1 &lt; len(data):
      current_data = data[start:end]
      start_timeout(time=TIMEOUT) # which would be the best way to to do that without breaking multiprocessing?
      try:
          with Pool() as p:
              for _ in p.imap_unordered(function, current_data):
                  save_checkpoint()
                  log()
                  start += 1
                  end += 1
      except Timeout:
          pass
      
  </code></pre>
  <p>Or any suggestion you think would be better. Any help would be much appreciated, thanks!</p>