title: multiprocessing: sharing a large read-only object between processes?
body: <p>Do child processes spawned via <a href="http://docs.python.org/library/multiprocessing.html" rel="noreferrer">multiprocessing</a> share objects created earlier in the program?</p>
  
  <p>I have the following setup:</p>
  
  <pre><code>do_some_processing(filename):
      for line in file(filename):
          if line.split(',')[0] in big_lookup_object:
              # something here
  
  if __name__ == '__main__':
      big_lookup_object = marshal.load('file.bin')
      pool = Pool(processes=4)
      print pool.map(do_some_processing, glob.glob('*.data'))
  </code></pre>
  
  <p>I'm loading some big object into memory, then creating a pool of workers that need to make use of that big object. The big object is accessed read-only, I don't need to pass modifications of it between processes.</p>
  
  <p>My question is: is the big object loaded into shared memory, as it would be if I spawned a process in unix/c, or does each process load its own copy of the big object? </p>
  
  <p>Update: to clarify further - big_lookup_object is a shared lookup object. I don't need to split that up and process it separately. I need to keep a single copy of it. The work that I need to split it is reading lots of other large files and looking up the items in those large files against the lookup object.</p>
  
  <p>Further update: database is a fine solution, memcached might be a better solution, and file on disk (shelve or dbm) might be even better. In this question I was particularly interested in an in memory solution. For the final solution I'll be using hadoop, but I wanted to see if I can have a local in-memory version as well.</p>