title: How would I use Dask to perform parallel operations on slices of NumPy arrays?
body: <p>I have a numpy array of coordinates of size n_slice x 2048 x 3, where n_slice is in the tens of thousands.  I want to apply the following operation on each 2048 x 3 slice separately</p>
  
  <pre><code>import numpy as np
  from scipy.spatial.distance import pdist
  
  # load coor from a binary xyz file, dcd format
  
  n_slice, n_coor, _ = coor.shape
  r = np.arange(n_coor)
  dist = np.zeros([n_slice, n_coor, n_coor])
  
  # this loop is what I want to parallelize, each slice is completely independent
  for i in xrange(n_slice): 
      dist[i, r[:, None] &lt; r] = pdist(coor[i])
  </code></pre>
  
  <p>I tried using Dask by making <code>coor</code> a <code>dask.array</code>,</p>
  
  <pre><code>import dask.array as da
  dcoor = da.from_array(coor, chunks=(1, 2048, 3))
  </code></pre>
  
  <p>but simply replacing <code>coor</code> by <code>dcoor</code> will not expose the parallelism.  I could see setting up parallel threads to run for each slice but how do I leverage Dask to handle the parallelism?</p>
  
  <p>Here is the parallel implementation using <code>concurrent.futures</code></p>
  
  <pre><code>import concurrent.futures
  import multiprocessing
  
  n_cpu = multiprocessing.cpu_count()
  
  def get_dist(coor, dist, r):
      dist[r[:, None] &lt; r] = pdist(coor)
  
  # load coor from a binary xyz file, dcd format
  
  n_slice, n_coor, _ = coor.shape
  r = np.arange(n_coor)
  dist = np.zeros([n_slice, n_coor, n_coor])
  
  with concurrent.futures.ThreadPoolExecutor(max_workers=n_cpu) as executor:
      for i in xrange(n_slice):
          executor.submit(get_dist, cool[i], dist[i], r)
  </code></pre>
  
  <p>It is possible this problem is not well suited to Dask since there are no inter-chunk computations.</p>