title: Alternative use patterns for python multiprocessing avoiding proliferation of global state?
body: <p>This (enormously simplified example) works fine (Python 2.6.6, Debian Squeeze):</p>
  
  <pre><code>from multiprocessing import Pool
  import numpy as np
  
  src=None
  
  def process(row):
      return np.sum(src[row])
  
  def main():
      global src
      src=np.ones((100,100))
  
      pool=Pool(processes=16)
      rows=pool.map(process,range(100))
      print rows
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>however, after years of being taught <strong><em>global state bad!!!</em></strong>, all my instincts are telling me I really really would rather be writing something closer to:</p>
  
  <pre><code>from multiprocessing import Pool
  import numpy as np
  
  def main():
      src=np.ones((100,100))
  
      def process(row):
          return np.sum(src[row])
  
      pool=Pool(processes=16)
      rows=pool.map(process,range(100))
      print rows
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>but of course that doesn't work (hangs up unable to pickle something).</p>
  
  <p>The example here is trivial, but by the time you add multiple "process" functions, and each of those is dependent on multiple additional inputs... well it all becomes a bit reminiscent of something written in BASIC 30 years ago.  Trying to use classes to at least aggregate the state with the appropriate functions seems an obvious solution, but <a href="https://stackoverflow.com/q/1816958/24283">doesn't seem to be that easy</a> in practice.</p>
  
  <p>Is there some recommended pattern or style for using multiprocessing.pool which will avoid the proliferation of global state to support each function I want to parallel map over ?</p>
  
  <p>How do experienced "multiprocessing pros" deal with this ?</p>
  
  <p><strong>Update</strong>: Note that I'm actually interested in processing much bigger arrays, so variations on the above which pickle <code>src</code> each call/iteration aren't nearly as good as ones which fork it into the pool's worker processes.</p>