title: Python Multiprocessing Pool as Decorator
body: <p>I'm working on code where I frequently have to use python's <code>multiprocessing</code> <code>Pool</code> class. This results in a ton of code that looks like this:</p>
  <pre><code>import time
  from multiprocessing import Pool
  from functools import partial
  
  def test_func(x):
      time.sleep(1)
      return x
  
  def test_func_parallel(iterable, processes):
      p = Pool(processes=processes)
      output = p.map(test_func, iterable)
      p.close()
      return output
  
  </code></pre>
  <p>This can be made more general:</p>
  <pre><code>def parallel(func, iterable, **kwargs):
      func = partial(func, **kwargs)
      p = Pool(processes=6)
      out = p.map(func, iterable)
      p.close()
      return out
  </code></pre>
  <p>This works, but adding a parallel wrapper to every other function complicates the code. What I'd really like is to get this working as a decorator. Something like this:</p>
  <pre><code>def parallel(num_processes):
      def parallel_decorator(func, num_processes=num_processes):
          def parallel_wrapper(iterable, **kwargs):
              func = partial(func, **kwargs)
              p = Pool(processes=num_processes)
              output = p.map(func, iterable)
              p.close()
              return output
  
          return parallel_wrapper
      return parallel_decorator
  </code></pre>
  <p>Which could be used as follows:</p>
  <pre><code>@parallel(6)
  def test_func(x):
      time.sleep(1)
      return x
  </code></pre>
  <p>This fails for pickle reasons</p>
  <p><code>Can't pickle &lt;function test1 at 0x117473268&gt;: it's not the same object as __main__.test1</code></p>
  <p>I've read a few posts on related issues, but they all implement a solution where the multiprocessing is executed outside the decorator. Does anyone know a way to make this work?</p>