title: Share Large, Read-Only Numpy Array Between Multiprocessing Processes
body: <p>I have a 60GB SciPy Array (Matrix) I must share between 5+ <code>multiprocessing</code> <code>Process</code> objects. I've seen numpy-sharedmem and read <a href="http://grokbase.com/t/python/python-list/1144s75ps4/multiprocessing-shared-memory-vs-pickled-copies">this discussion</a> on the SciPy list. There seem to be two approaches--<code>numpy-sharedmem</code> and using a <code>multiprocessing.RawArray()</code> and mapping NumPy <code>dtype</code>s to <code>ctype</code>s. Now, <code>numpy-sharedmem</code> seems to be the way to go, but I've yet to see a good reference example. I don't need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I'd like to avoid a copy. It <em>sounds like</em> the correct method is to create the <em>only</em> copy of the array as a <code>sharedmem</code> array, and then pass it to the <code>Process</code> objects? A couple of specific questions:</p>
  
  <ol>
  <li><p>What's the best way to actually pass the sharedmem handles to sub-<code>Process()</code>es? Do I need a queue just to pass one array around? Would a pipe be better? Can I just pass it as an argument to the <code>Process()</code> subclass's init (where I'm assuming it's pickled)?</p></li>
  <li><p>In the discussion I linked above, there's mention of <code>numpy-sharedmem</code> not being 64bit-safe? I'm definitely using some structures that aren't 32-bit addressable.</p></li>
  <li><p>Are there tradeoff's to the <code>RawArray()</code> approach? Slower, buggier?</p></li>
  <li><p>Do I need any ctype-to-dtype mapping for the numpy-sharedmem method?</p></li>
  <li><p>Does anyone have an example of some OpenSource code doing this? I'm a very hands-on learned and it's hard to get this working without any kind of good example to look at.</p></li>
  </ol>
  
  <p>If there's any additional info I can provide to help clarify this for others, please comment and I'll add. Thanks!</p>
  
  <p>This needs to run on Ubuntu Linux and <em>Maybe</em> Mac OS, but portability isn't a huge concern.</p>