title: Best practice for loading large dataset and using dask.delayed
body: <p>I have a <code>csv</code> file of 550,000 rows of text. I read it into a pandas dataframe, loop over it, and perform some operation on it. Here is some sample code:</p>
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  
  def my_operation(row_str):
     #perform operation on row_str to create new_row_str
     return new_row_str
  
  df = pd.read_csv('path/to/myfile.csv')
  results_list = []
  for ii in range(df.shape[0]):
     my_new_str = my_operation(df.iloc[ii, 0])
     results_list.append(my_new_str)
  </code></pre>
  <p>I started to implement <code>dask.delayed</code> but after reading the <a href="https://docs.dask.org/en/latest/delayed-best-practices.html" rel="nofollow noreferrer">Delayed Best Practices section</a>, I am not sure I am using <code>dask.delayed</code> in the most optimal way for this problem. Here is the same code with <code>dask.delayed</code>:</p>
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  import dask
  
  def my_operation(row_str):
     #perform operation on row_str to create new_row_str
     return new_row_str
  
  df = pd.read_csv('path/to/myfile.csv')
  results_list = []
  for ii in range(df.shape[0]):
     my_new_str = dask.delayed(my_operation)(df.iloc[ii, 0])
     results_list.append(my_new_str)
  
  results_list = dask.compute(*results_list)
  </code></pre>
  <p>I'm running this on a single machine with 8 cores and was wanting to know if there was a more optimal way to load this large dataset and perform the same operation over each of the rows?</p>
  <p>Thanks in advance for your help and let me know what else I can provide!</p>