title: numpy vs. multiprocessing and mmap
body: <p>I am using Python's <code>multiprocessing</code> module to process large numpy arrays in parallel. The arrays are memory-mapped using <code>numpy.load(mmap_mode='r')</code> in the master process. After that, <code>multiprocessing.Pool()</code> forks the process (I presume).</p>
  
  <p>Everything seems to work fine, except I am getting lines like:</p>
  
  <pre class="lang-none prettyprint-override"><code>AttributeError("'NoneType' object has no attribute 'tell'",)
    in `&lt;bound method memmap.__del__ of
         memmap([ 0.57735026,  0.57735026,  0.57735026,  0.        ,  0.        ,        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,        0.        ,  0.        ], dtype=float32)&gt;`
       ignored
  </code></pre>
  
  <p>in the unittest logs. The tests pass fine, nevertheless.</p>
  
  <p>Any idea what's going on there?</p>
  
  <p>Using Python 2.7.2, OS X, NumPy 1.6.1.</p>
  
  <hr>
  
  <p>UPDATE:</p>
  
  <p>After some debugging, I hunted down the cause to a code path that was using a (small slice of) this memory-mapped numpy array as input to a <code>Pool.imap</code> call.</p>
  
  <p>Apparently the "issue" is with the way <code>multiprocessing.Pool.imap</code> passes its input to the new processes: it uses pickle. This doesn't work with <code>mmap</code>ed numpy arrays, and something inside breaks which leads to the error.</p>
  
  <p>I found <a href="http://old.nabble.com/Annoyance-of-memap-rray-with-multiprocessing.Pool.applay_async-td28129361.html" rel="nofollow noreferrer">this reply</a> by Robert Kern which seems to address the same issue. He suggests creating a special code path for when the <code>imap</code> input comes from a memory-mapped array: memory-mapping the same array manually in the spawned process.</p>
  
  <p>This would be so complicated and ugly that I'd rather live with the error and the extra memory copies. Is there any other way that would be lighter on modifying existing code?</p>