title: Share large data between processes within a class
body: <p>I'm trying to share a large numpy array between processes using pool.imap_unordered. Should be easy but I'm trying to do it from within a <em>class</em>. Right now I'm just passing the data everytime and everything works well until the data gets sufficiently large and pool just hangs and doesn't launch the parallel processes. Since only a subset of the large data is needed for each parallel process, an alternative is to only pass a subset, but I don't know how to in my current framework.</p>
  
  <p>Since functions used for multiprocessing must be in the global namespace, I'm placing my function outside of the class as follows (toy example of real problem):</p>
  
  <pre><code>import numpy as np
  import mutliprocessing.Pool
  import itertools
  
  def process(args):
      large_data, index = args
      return some_costly_operation(large_data[index])
  
  class MyClass:
      def __init__(self):
          # Let's pretend this is large
          self.data = np.zeros(10)
  
      def do(self):
          p = Pool()
          for result in p.imap_unordered(process,
              itertools.izip(itertools.repeat(self.data), xrange(10)))):
  
              print result
  </code></pre>
  
  <p>I know this is a hack-y way to do multiprocessing and theoretically you shouldn't do it from within a class and should protect by checking if you're in main... Any alternatives or suggestions?</p>