<p>I have set up a Kubernetes cluster using Kubernetes Engine on GCP to work on some data preprocessing and modelling using Dask. I installed Dask using Helm <a href="http://docs.dask.org/en/latest/setup/kubernetes-helm.html" rel="nofollow noreferrer">following these instructions</a>.</p>

<p>Right now, I see that there are two folders, <code>work</code> and <code>examples</code></p>

<p><a href="https://i.stack.imgur.com/7wIcs.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7wIcs.png" alt="enter image description here"></a></p>

<p>I was able to execute the contents of the notebooks in the <code>example</code> folder confirming that everything is working as expected.</p>

<p>My questions now are as follows</p>

<ul>
<li>What are the suggested workflow to follow when working on a cluster? Should I just create a new notebook under <code>work</code> and begin prototyping my data preprocessing scripts? </li>
<li>How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment? Would you just manually move them to a bucket every time you upgrade (which seems tedious)? or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?</li>
</ul>

<p>I'm new to working with data in a distributed environment in the cloud so any suggestions are welcome.</p>

PollPenn
<blockquote>
  <p>What are the suggested workflow to follow when working on a cluster? </p>
</blockquote>

<p>There are many workflows that work well for different groups.  There is no single blessed workflow.</p>

<blockquote>
  <p>Should I just create a new notebook under work and begin prototyping my data preprocessing scripts?</p>
</blockquote>

<p>Sure, that would be fine.</p>

<blockquote>
  <p>How can I ensure that my work doesn't get erased whenever I upgrade my Helm deployment? </p>
</blockquote>

<p>You might save your data to some more permanent store, like cloud storage, or a git repository hosted elsewhere.</p>

<blockquote>
  <p>Would you just manually move them to a bucket every time you upgrade (which seems tedious)? </p>
</blockquote>

<p>Yes, that would work (and yes, it is)</p>

<blockquote>
  <p>or would you create a simple vm instance, prototype there, then move everything to the cluster when running on the full dataset?</p>
</blockquote>

<p>Yes, that would also work.</p>

<h3>In Summary</h3>

<p>The Helm chart includes a Jupyter notebook server for convenience and easy testing, but it is no substitute for a full fledged long-term persistent productivity suite.  For that you might consider a project like JupyterHub (which handles the problems you list above) or one of the many enterprise-targeted variants on the market today.  It would be easy to use Dask alongside any of those.</p>
