<p>We are trying to make some analysis code spin on the kubernetes cluster. We want to run 10 pods of 1 job with the following yaml file:</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
 name: job-platooning-s1b
spec:
 parallelism: 10
 template:
   metadata:
     name: job-platooning-s1b
   spec:
     containers:
     - name: platooning-dp-1b
       image: registry.gitlab.com/company_name/repo_name/platooning:latest
       command: ["python3" , "/app/scenario_1b_cluster.py"]
     restartPolicy: 'OnFailure'
     imagePullSecrets:
     - name: regcred-nextsys
</code></pre>

<p>Our 10 pods can survive for a few minutes before getting killed. The error that I get is: <code>MountVolume.SetUp failed for volume "default-token-7td4s" : couldn't propagate object cache: timed out waiting for the condition</code>.</p>

<p>My thinking is that the pods consume too much memory. We tried to specify memory usage by adding the following parameters under <code>containers</code> in the yaml file:</p>

<pre><code>resources:
    limits:
        memory: "15Gi"
    requests:
        memory: "500Mi"
</code></pre>

<p>But it does not help, as pods are still terminated. Running the job with 1 pod is fine, as it does not get killed. In the end, we want to have a scalable solution where multiple scenarios with multiple pods could be run overnight.</p>

<p>Do you have any idea about why the pods are getting killed in this scenario?</p>

<p>The pods are running properly when they run one by one (without parallelism). When we try to run a lot of them together, they run for a bit then get killed (evicted?) sometimes producing this error : </p>

<blockquote>
  <p>MountVolume.SetUp failed for volume "default-token-7td4s" : couldn't propagate object cache: timed out waiting for the condition</p>
</blockquote>

<p>The strange thing is that we the secret this job uses is not default-token-7td4s but regcred-nextsys, as seen in the job YAML file. Is that expected behaviour? And if so, why does it actually fail? I'm suspecting a race condition, or just different pods trying to mount the same resource, but I'm not sure that it makes sense. The other reason I suspect is a memory issue.</p>

<p>We are running kubernetes as a managed service from DigitalOcean.</p>

Pavlo Bazilnskyy
<p>I'm working together with Pavlo and encountered this issue. What I understand from now is that the pods are running properly when they run one by one (without parallelism). When we try to run a lot of them together, they run for a bit then get killed (evicted?) sometimes producing this error : </p>

<blockquote>
  <p>MountVolume.SetUp failed for volume "default-token-7td4s" : couldn't propagate object cache: timed out waiting for the condition</p>
</blockquote>

<p>The strange thing is that we the secret this job uses is not default-token-7td4s but regcred-nextsys, as seen in the job YAML file. Is that expected behaviour? And if so, why does it actually fail? I'm suspecting a race condition, or just different pods trying to mount the same resource, but I'm not sure that it makes sense. The other reason I suspect is a memory issue.</p>

<p>We are running kubernetes as a managed service from DigitalOcean.</p>
