title: When to and when not to use map() with multiprocessing.Pool, in Python? case of big input values
body: <p>Is it efficient to calculate many results in parallel with <code>multiprocessing.Pool.map()</code> in a situation where each input value is <em>large</em> (say 500 MB), but where input values general contain the same large object?  I am afraid that the way <code>multiprocessing</code> works is by sending a pickled version of each input value to each worker process in the pool.  If no optimization is performed, this would mean sending a lot of data for each input value in <code>map()</code>.  Is this the case?  I quickly had a look at the <code>multiprocessing</code> code but did not find anything obvious.</p>
  
  <p>More generally, what simple parallelization strategy would you recommend so as to do a <code>map()</code> on say 10,000 values, each of them being a tuple <code>(vector, very_large_matrix)</code>, where the vectors are always different, but where there are say only 5 different very large matrices?</p>
  
  <p><strong>PS</strong>: the big input matrices actually appear "progressively": 2,000 vectors are first sent along with the <em>first</em> matrix, then 2,000 vectors are sent with the second matrix, etc.</p>