<p>I'm attempting to deploy a parallelized data processing task that uses many containers of the same docker image which would each run with different incrementing environment variables. The image is setup to read env vars in order to determine which segment of a larger list to process.</p>

<p><strong>Background:</strong> I was originally using a bash script that passed down an incrementing env var to docker run commands but now I'd like a better way to manage/monitor all the containers. I've only had experience using Kubernetes for application services but it seems like it may be a better way to orchestrate my multi-container task as well.</p>

<p>Wondering if this sort of dynamic environment variable passing is possible within the Kubernetes YAML configs as I'd prefer declarative config vs a shell script. I'm also not sure of the best approach to do this in Kubernetes whether it's multiple separate pods, multi-container pod, or to use replicas in some way.</p>

<p>I'm open to suggestions, I know other tools like Terraform may be helpful for this sort of programatic infrastructure also.</p>

ev-dev
<p>What about using <a href="https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/" rel="nofollow noreferrer">Parallel Processing Using Work Queue</a> for passing different environment variables to your k8s job pods with .spec.parallelism. Although having a separate service for work queue may be little too much depending on what you are trying to do.</p>

<p>The other idea can be using helm templating power to create a k8s manifest file. I create a sample helm chart to give an idea of templating parallel processing. See <a href="https://github.com/jayrajput/helm-parallel-jobs" rel="nofollow noreferrer">git repo - helm-parallel-jobs</a>. Once you have the git repo cloned, you can install the helm chart for parallel processing like this. The template for job is same as used by <a href="https://kubernetes.io/docs/tasks/job/parallel-processing-expansion/" rel="nofollow noreferrer">k8s documentation</a>. As seen in the output below, three different environment variables - apple,banana,cherry are provided, which creates 3 different pods with environment variables passed to them.</p>

<pre><code>    [root@jr]# helm install --set envs='{apple,banana,cherry}'  --name jobs ./helm-parallel-jobs/example/parallel-jobs
    NAME:   jobs
    LAST DEPLOYED: Sun Aug 26 16:29:23 2018
    NAMESPACE: default
    STATUS: DEPLOYED

    RESOURCES:
    ==&gt; v1/Job
    NAME                 DESIRED  SUCCESSFUL  AGE
    process-item-apple   1        0           0s
    process-item-banana  1        0           0s
    process-item-cherry  1        0           0s

    ==&gt; v1/Pod(related)
    NAME                       READY  STATUS             RESTARTS  AGE
    process-item-apple-dr6st   0/1    ContainerCreating  0         0s
    process-item-banana-d2wwq  0/1    ContainerCreating  0         0s
    process-item-cherry-wvlxz  0/1    ContainerCreating  0         0s
</code></pre>
