<p>I am running a job as defined <a href="https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/" rel="nofollow noreferrer">here</a> i.e. "parallel processing using work queue"
on GKE</p>

<p>Each pod runs a single container and That container has an R script, takes around 5 mins to finish and then Pod completes successfully.</p>

<p>When I run the job for small numbers of completions like</p>

<pre><code>completions: 606
parallelism: 450
backoffLimit: 1
</code></pre>

<p>Everything runs fine, Cluster scales up and down properly and Job gets finished.</p>

<p>But when I run the job with specifications like</p>

<pre><code>completions: 37572
parallelism: 1610
backoffLimit: 1
</code></pre>

<blockquote>
  <p>Pod succeeded count gets increased for a while but after that, it stays
  in between 1000 to 1500 and never reach to completions</p>
</blockquote>

<p><a href="https://i.stack.imgur.com/PsOlR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PsOlR.png" alt="enter image description here"></a></p>

<p>Though pods are completing successfully, I can see them on google cloud kubernetes dashboard and output files are also being generated successfully.
Queue is also showing the progress very accurately
<a href="https://i.stack.imgur.com/M7JfT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/M7JfT.png" alt="enter image description here"></a></p>

<p>And It has happened to me every time when I have run the job with high number of parallelism.
I have tried different autoscaling node pool setup in my cluster for machine type with 64-CPUs, 32-CPUs, 16-CPUs. </p>

<p>Currently, the way I am handling is<br>
=> When queue has number of consumers == parallelism OR files_in_output == completions<br>
=> I delete the job and delete the autoscaling node pool.</p>

<p>Please find the cluster details
<a href="https://i.stack.imgur.com/zvCH0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zvCH0.png" alt="enter image description here"></a></p>

<p>Cluster status is always green during the run.</p>

<p><strong>QUESTION</strong>         </p>

<ul>
<li>Why job completion count never increase after a certain point (i.e in my case below my parallelism count)? Even though pods are finishing successfully.</li>
<li>And the worst thing is the job completion count decreases too? That I can't even understand. What will be the reason that kubernetes is behaving like so?</li>
<li>Do I need to add some additional fields to my spec template so that
it tracks job completions properly?</li>
</ul>

<p>Update:</p>

<ul>
<li>I have enough CPU Quota</li>
<li>Each container(Pod) is limited to use 1 CPU and 1GB RAM.</li>
<li>I have also upgraded cluster and node-pools to 1.10.6-gke.2 version. No Luck.</li>
</ul>

<p>GKE ISSUE REPORTED => <a href="https://issuetracker.google.com/issues/114650730" rel="nofollow noreferrer">https://issuetracker.google.com/issues/114650730</a></p>

<p><strong>job.yml</strong></p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  # Unique key of the Job instance
  name: my-job
spec:
  completions: 37572
  parallelism: 1610
  backoffLimit: 1
  template:
    metadata:
      name: my-job
      labels:
        jobgroup: my-jobs
    spec:
      volumes:
      - name: jobs-pv-storage
        persistentVolumeClaim:
          claimName: fileserver-claim
          readOnly: false
      containers:
      - name: rscript
        image: gcr.io/project/image:v1
        resources:
          limits:
            cpu: "1"
            memory: 1200Mi
          requests:
            cpu: "1"
            memory: 1000Mi
        env:
        - name: BROKER_URL
          value: amqp://user:pwd@rabbitmq-service:5672
        - name: QUEUE
          value: job-queue
        volumeMounts:
        - mountPath: /opt/work/output
          name: jobs-pv-storage
      # Do not restart containers after they exit
      restartPolicy: Never
</code></pre>

<p>Posting as Community Wiki.</p>
<p>As mentioned by @Rahul Gautam original issue: <a href="https://issuetracker.google.com/issues/114650730" rel="nofollow noreferrer">https://issuetracker.google.com/issues/114650730</a> is not visible as it contains some Private information. There is a new public issue: <a href="https://issuetracker.google.com/issues/172663707" rel="nofollow noreferrer">https://issuetracker.google.com/issues/172663707</a> which was closed as <code>Intended behavior</code></p>
<p>In nutshell, this scenario is related with Pod Garbage Collector behavior.</p>
<p>Pod Garbage Collector interferes Job completion. Garbage Collector threshold is controlled by <code>--terminated-pod-gc-threshold</code> flag in <code>kube-controller-manager</code> as documented <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="nofollow noreferrer">here</a>. Default value is <code>12500</code> and it's cluster wide so it counts all jobs in all namespaces.</p>
<pre><code>--terminated-pod-gc-threshold int32     Default: 12500
Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If &lt;= 0, the terminated pod garbage collector is disabled.
</code></pre>
<p>Due to that fact, when number of jobs are lower than 12500 you will not experience this issue.</p>
<p>As a workaround you could change <code>--terminated-pod-gc-threshold</code> flag to not hit this threshold. However, in GKE controller manager is managed by Google, thus GKE users cannot change that value.</p>
<p>In addition, there is already an open <a href="https://github.com/kubernetes/kubernetes/issues/28486" rel="nofollow noreferrer">Github thread</a> about re-implementing this behavior.</p>
