<p>I'm attempting to deploy a parallelized data processing task that uses many containers of the same docker image which would each run with different incrementing environment variables. The image is setup to read env vars in order to determine which segment of a larger list to process.</p>

<p><strong>Background:</strong> I was originally using a bash script that passed down an incrementing env var to docker run commands but now I'd like a better way to manage/monitor all the containers. I've only had experience using Kubernetes for application services but it seems like it may be a better way to orchestrate my multi-container task as well.</p>

<p>Wondering if this sort of dynamic environment variable passing is possible within the Kubernetes YAML configs as I'd prefer declarative config vs a shell script. I'm also not sure of the best approach to do this in Kubernetes whether it's multiple separate pods, multi-container pod, or to use replicas in some way.</p>

<p>I'm open to suggestions, I know other tools like Terraform may be helpful for this sort of programatic infrastructure also.</p>

ev-dev
<p>My understanding is you'd like to do something like <a href="https://kubernetes.io/docs/tasks/job/parallel-processing-expansion/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/job/parallel-processing-expansion/</a> where jobs are created from a template, one for each data item in a list. But you don't want it to be shell-scripted. </p>

<p>I imagine helm could be used to replace the Job and it has a range function so a chart could be set up to create jobs for each entry in a section of a values.yaml. So it could occupy a space similar to what you suggested for terraform. Ansible could also be an option.  </p>

<p>However, the direction of travel of this question seems to be towards batch scheduling. I am wondering if your jobs will evolve to end up having dependencies between them etc. If so <a href="https://stackoverflow.com/questions/45454130/helm-and-kubernetes-is-there-a-barrier-equivalent-for-jobs">Helm and Kubernetes: Is there a barrier equivalent for jobs?</a> and <a href="https://www.quora.com/Is-Kubernetes-suited-for-long-running-batch-jobs" rel="nofollow noreferrer">https://www.quora.com/Is-Kubernetes-suited-for-long-running-batch-jobs</a> help here. Currently Kubernetes has facilities for running batch jobs and the tooling to enable a batch scheduling system to run or be built on it but it doesn't itself contain an out of the box batch scheduling system. So people are currently using a range of different approaches to suit their needs.  </p>
