<p>I need to process multiple files in s3 with k8s, so I've created one job on k8s and contains approximately 500 containers, for each container with different envs. However, the job is very slowly and in multiple times it failed.</p>
<p>I'm using kubernetes api python to submit job, likes this:</p>
<pre><code>def read_path():
    files = []
    suffixes = ('_SUCCESS', 'referen/')

    files_path = fs.listdir(f&quot;s3://{path}&quot;)
    if files_path is not None:
        for num, file in enumerate(files_path, start=1):
            if not file['Key'].endswith(suffixes):
                files.append(file['Key'])
    return files

def from_containers(container, path):
    containers = []
    for num, file in enumerate(read_path(container, path), start=1):
        containers.append(client.V1Container(name=f'hcm1-{num}', image='image-python',
                                             command=['python3', 'model.py'],
                                             env=[client.V1EnvVar(name='model', value='hcm1'),
                                                  client.V1EnvVar(name='referen', value=f&quot;s3://{file}&quot;)]))


template = client.V1PodTemplateSpec(metadata=client.V1ObjectMeta(name=&quot;hcm1&quot;),
                                    spec=client.V1PodSpec(restart_policy=&quot;OnFailure&quot;,
                                    containers=from_containers(containers, path),
                                    image_pull_secrets=[client.V1LocalObjectReference(name=&quot;secret&quot;)]))

spec = client.V1JobSpec(template=template, backoff_limit=20)

job = client.V1Job(api_version=&quot;batch/v1&quot;, kind=&quot;Job&quot;, metadata=client.V1ObjectMeta(name=&quot;hcm1&quot;), spec=spec)

api_response = batch_v1.create_namespaced_job(body=job, namespace=&quot;default&quot;)
print(&quot;Job created. status='%s'&quot; % str(api_response.status))

</code></pre>
<p>I'm tried to use some config, like completions=10, concurrency=2, but my job multiple number of times to execute, 10*500=5000</p>
<p>What's better way to create job on k8s with multiples containers?</p>
<ol>
<li>1 Job -&gt; 1 Pod -&gt; 500 containers</li>
<li>1 Job -&gt; 500 Pods (Each pod 1 container).</li>
<li>Is there another way?**</li>
</ol>

Alan Miranda
<p>Your question is slightly <a href="https://english.meta.stackexchange.com/a/6528">opinion based</a>. You can find many solutions depending on your situation and approach.
However <a href="https://stackoverflow.com/users/10008173/david-maze">David Maze</a> mentioned very well:</p>
<blockquote>
<p>If you have this scripting setup already, why not 500 Jobs? (Generally you want to avoid multiple containers per pod.)</p>
</blockquote>
<p>Yes, if you are writing a script, this could be a very good solution to the problem. It definitely won't be that complicated because create 1 <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/" rel="nofollow noreferrer">Job</a> for 1 pod. There will be 1 container in each pod.</p>
<p>Theoretically you can also define your own <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/" rel="nofollow noreferrer">Custom Resource</a> and write a controller like Job controller which supports multiple pod templates.</p>
<p>See also <a href="https://stackoverflow.com/questions/63160405/can-a-single-kubernetes-job-contain-multiple-pods-with-different-parallelism-def">this question.</a></p>
