<p>I have a gce airflow (composer) cluster with a bunch of workers:</p>

<pre><code>$ kubectl get pods
NAME                                 READY     STATUS    RESTARTS   AGE
airflow-redis-0                      1/1       Running   0          7h
airflow-scheduler                    2/2       Running   0          7h
airflow-sqlproxy                     1/1       Running   0          8h
airflow-worker                       50/50     Running   0          7h
composer-fluentd-daemon              1/1       Running   0          7h
composer-fluentd-daemon              1/1       Running   0          7h
</code></pre>

<p>I also have a bunch of unique persistent NFS volumes that have data that need processing.  Is there a way to dynamically mount a different NFS volume to each of the respective workers.</p>

<p>Alternatively, is it possible for the DockerOperator called within the worker to mount the NFS volume pertaining to its specific workload.</p>

<p>In theory the workflow would be: <code>Spin up 1x worker per Dataset</code> > <code>Get Dataset</code> > <code>Run Dataset through Model</code> > <code>Dump results</code></p>

<p>One way to accomplish this would be to download the Dataset to the given pod that is processing it; however, these Datasets are several hundred gb per and will need to be processed many times against different models.</p>

<p>Eventually we plan on putting all of this data in BigTable, but I need to show a proof in concept using volumes with a few hundred gb of data before we get the green light to spin up a BigTable cluster with multiple tb of data in it.</p>

<p>Input appreciated.  Telling me im doing it wrong with a better solution is also a viable answer.</p>

<p>Deployment, by definition, uses a set of identical replicas as pods (i.e. ReplicaSet). Therefore all pods of a deployment will have the PodSpec, pointing to the same volume.</p>

<p>Sounds like you need to write some custom logic yourself to orchestrate spinning up new workloads (i.e. Jobs) with different volumes.</p>

<p>You can do this by simply deploying a bash script that calls into kubectl (by default, kubectl inside a pod can work directly) in a loop. Or you can write something that uses Kubernetes API and makes some API calls to discover the new volumes, create workloads to process them (and then maybe clean up the volumes).</p>
