title: Passing shared memory variables in python multiprocessing
body: <p>I have a bunch of files that I want to read in parallel using Python's multiprocessing and collect all the data in a single NumPy array. For this purpose, I want to define a shared memory NumPy array and pass its slices to different processes to read in parallel. A toy illustration of what I am trying to do is given in the following code where I am trying to modify a numpy array using multiprocessing.</p>
  <p>Example 1:</p>
  <pre><code>
  import numpy as np
  import multiprocessing
  
  def do_stuff(i, arr):
      arr[:]=i
      return
  
  def print_error(err):
      print(err)
  
  if __name__ == '__main__':
      idx = [0,1,2,3]
      # Need to fill this array in parallel
      arr = np.zeros(4)
      p = multiprocessing.Pool(4)
      # Passing slices to arr to modify using multiprocessing
      for i in idx:
          p.apply(do_stuff, args=(i,arr[i:i+1]))
      p.close()
      p.join()
      print(arr)
  
  </code></pre>
  <p>In this code, I want the arr to be filled with 0, 1, 2, 3. This however prints arr to be all zeros. After reading the answers <a href="https://stackoverflow.com/a/5550156/7656776">here</a>, I used multiprocessing.Array to define the shared memory variable and modified my code as follows</p>
  <p>Example 2:</p>
  <pre><code>import numpy as np
  import multiprocessing
  
  def do_stuff(i, arr):
      arr[:]=i
      return
  
  def print_error(err):
      print(err)
  
  if __name__ == '__main__':
      idx = [0,1,2,3]
      p = multiprocessing.Pool(4)
      # Shared memory Array
      shared = multiprocessing.Array('d', 4)
      arr = np.ctypeslib.as_array(shared.get_obj())
  
      for i in idx:
          p.apply(do_stuff, args=(i,arr[i:i+1]))
      p.close()
      p.join()
      print(arr)
  
  </code></pre>
  <p>This also prints all zeros for arr. However, when I define the array outside main and use pool.map, the code works. For e.g., the following code works</p>
  <p>Example 3:</p>
  <pre><code>import numpy as np
  import multiprocessing
  
  shared = multiprocessing.Array('d', 4)
  arr = np.ctypeslib.as_array(shared.get_obj())
  
  def do_stuff(i):
      arr[i]=i
      return
  
  def print_error(err):
      print(err)
  
  if __name__ == '__main__':
      idx = [0,1,2,3]
      p = multiprocessing.Pool(4)
      shared = multiprocessing.Array('d', 4)
      p.map(do_stuff, idx)
      p.close()
      p.join()
      print(arr)
               
  </code></pre>
  <p>This prints [0,1,2,3].</p>
  <p>I am very confused by all this. My questions are:</p>
  <ol>
  <li><p>When I define arr = np.zeros(4), which processor owns this variable? When I then send the slice of this array to different processors what is being sent if this variable is not defined on those processors.</p>
  </li>
  <li><p>Why doesn't example 2 work while example 3 does?</p>
  </li>
  </ol>
  <p>I am working on Linux and Python/3.7/4</p>