title: Dask delayed / dask array no response
body: <p>I have a distributed dask cluster setup and I have used it to load and transform a bunch of data. Works like a charm.</p>
  <p>I'm want to use it do some processing in parallel. Here's my function</p>
  <pre><code>el = 5000
  n_using = 26
  n_across= 6
  
  mat = np.random.random((el,n_using,n_across))
  idx = np.tril_indices(n_across*2, -n_across)
  
  def get_vals(c1, m, el, idx):
      m1 = m[c1,:,:]
      corr_vals = np.zeros((el, (n_across//2)*(n_across+1)))
      for c2 in range(c1+1, el):
          corr = np.corrcoef(m1.T, m[c2,:,:].T)
          corr_vals[c2] = corr[idx]
          
      return corr_vals
  
  lazy_get_val = dask.delayed(get_vals, pure=True)
  </code></pre>
  <p>Here is a single processor version of what I'm trying to do:</p>
  <pre><code>arrays = [get_vals(c1, mat, el, idx) for c1 in range(el)]
  all_corr = np.stack(arrays, axis=0)
  </code></pre>
  <p>Works fine but takes a few hours.
  Here's my go at doing this in dask:</p>
  <pre><code>lazy_list = [lazy_get_val(c1, mat, el, idx) for c1 in range(el)]
  arrays = [da.from_delayed(lazy_item, dtype=float, shape=(el, 21)) for lazy_item in lazy_list]
  all_corr = da.stack(arrays, axis=0)
  </code></pre>
  <p>Even if it run <code>all_corr[1].compute()</code>, it just sits there and doesn't respond. When I interrupt the kernel, it seems to be stuck at /distributed/utils.py:</p>
  <blockquote>
  <p>~/.../lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)</p>
  <pre><code>    249     else:
      250         while not e.is_set():
  --&gt; 251             e.wait(10)
      252     if error[0]:
      253         six.reraise(*error[0])
  </code></pre>
  </blockquote>
  <p>Any suggestions on debugging this?</p>
  <hr />
  <p>Other things:</p>
  <ul>
  <li>If I run it with a smaller <code>mat</code> (el=1000) and it runs fine.</li>
  <li>If I make <code>el = 5000</code>, it hangs.</li>
  <li>If I interrupt the kernel and run it again with <code>el = 1000</code>, it hangs.</li>
  </ul>