title: Shuffling data in dask
body: <p>This is a follow on question from <a href="https://stackoverflow.com/questions/46820223/subsetting-dask-dataframes">Subsetting Dask DataFrames</a>. I wish to shuffle data from a <strong>dask dataframe</strong> before sending it in batches to a ML algorithm. </p>
  
  <p>The answer in that question was to do the following:</p>
  
  <pre><code>for part in df.repartition(npartitions=100).to_delayed():
      batch = part.compute()
  </code></pre>
  
  <p>However, even if I was to shuffle the contents of batch I'm a bit worried that it might not be ideal. The data is a time series set so datapoints would be highly correlated within each partition.</p>
  
  <p>What I would ideally like is something along the lines of:</p>
  
  <pre><code>rand_idx = np.random.choice(len(df), batch_size, replace=False)
  batch = df.iloc[rand_idx, :]
  </code></pre>
  
  <p>which would work on pandas but not dask. Any thoughts?</p>
  
  <h2>Edit 1: Potential Solution</h2>
  
  <p>I tried doing</p>
  
  <pre><code>train_len = int(len_df*0.8)
  idx = np.random.permutation(len_df)
  train_idx = idx[:train_len]
  test_idx = idx[train_len:]
  train_df = df.loc[train_idx]
  test_df = df.loc[test_idx]
  </code></pre>
  
  <p>However, if I try doing <code>train_df.loc[:5,:].compute()</code> this return a 124451 row dataframe. So clearly using dask wrong.</p>