<p>I have a gce airflow (composer) cluster with a bunch of workers:</p>

<pre><code>$ kubectl get pods
NAME                                 READY     STATUS    RESTARTS   AGE
airflow-redis-0                      1/1       Running   0          7h
airflow-scheduler                    2/2       Running   0          7h
airflow-sqlproxy                     1/1       Running   0          8h
airflow-worker                       50/50     Running   0          7h
composer-fluentd-daemon              1/1       Running   0          7h
composer-fluentd-daemon              1/1       Running   0          7h
</code></pre>

<p>I also have a bunch of unique persistent NFS volumes that have data that need processing.  Is there a way to dynamically mount a different NFS volume to each of the respective workers.</p>

<p>Alternatively, is it possible for the DockerOperator called within the worker to mount the NFS volume pertaining to its specific workload.</p>

<p>In theory the workflow would be: <code>Spin up 1x worker per Dataset</code> > <code>Get Dataset</code> > <code>Run Dataset through Model</code> > <code>Dump results</code></p>

<p>One way to accomplish this would be to download the Dataset to the given pod that is processing it; however, these Datasets are several hundred gb per and will need to be processed many times against different models.</p>

<p>Eventually we plan on putting all of this data in BigTable, but I need to show a proof in concept using volumes with a few hundred gb of data before we get the green light to spin up a BigTable cluster with multiple tb of data in it.</p>

<p>Input appreciated.  Telling me im doing it wrong with a better solution is also a viable answer.</p>

<p>The workflow you describe better matches the model of a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="nofollow noreferrer">Job</a> than a plain (long-running) Pod.  You would need to create a separate job spec for each task pointing at its respective data, and if your cluster is doing other work, you would need to take care that your bulk-data-processing pods don't overwhelm the available compute resources.</p>

<p>Do you actually have distinct NFS volumes (server names/exported directories), or just many file trees in a single NFS volume?  If the latter, another path that could work well for you is to set up a queueing system like <a href="http://www.rabbitmq.com" rel="nofollow noreferrer">RabbitMQ</a> and load all of the paths into a queue there.  You would then write a long-running process that serially reads a single item off the queue, does whatever required work on it, writes its result, commits the work item, and repeats (in a single thread).  Then you'd scale that up using a Deployment to the desired amount of parallel processing.</p>

<p>In any case, your output suggests you're working directly with bare pods and have tried to scale the workers by having many parallel worker containers in a single pod.  It's better to use one of the higher-level controllers (most often a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="nofollow noreferrer">Deployment</a>) and use its <code>replicas:</code> control to launch multiple copies of the described pod.  Among other things, this will let you spread load across multiple nodes, and will let you do a rolling update where pods get incrementally restarted, avoiding an outage if you change the underlying image or other details of the pod.</p>
