<p>We are using the Job <code>ttlSecondsAfterFinished</code> attribute to automatically clean up finished jobs. When we had a very small number of jobs (10-50), the jobs (and their pods) would get cleaned up approximately 60 seconds after completion. However, now that we have ~5000 jobs running on our cluster, it takes 30 + minutes for a Job object to get cleaned after completion.</p>
<p>This is a problem because although the Jobs are just sitting there, not consuming resources, we do use a <code>ResourceQuota</code> (selector count/jobs.batch) to control our workload, and those completed jobs are taking up space in the <code>ResourceQuota</code>.</p>
<p>I know that jobs only get marked for deletion once the TTL has passed, and are not guaranteed to be deleted immediately then, but 30 minutes is a very long time. What could be causing this long delay?</p>
<p>Our logs indicate that our k8s API servers are not under heavy load, and that API response times are reasonable.</p>

<p><strong>Solution 1</strong></p>
<p>How do you use the Job <code>ttlSecondsAfterFinished</code>? You can specify <code>.spec.ttlSecondsAfterFinished</code> to the value what you need. Below is the example from <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs" rel="nofollow noreferrer">official documentation</a></p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: pi-with-ttl
spec:
  ttlSecondsAfterFinished: 100
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: [&quot;perl&quot;,  &quot;-Mbignum=bpi&quot;, &quot;-wle&quot;, &quot;print bpi(2000)&quot;]
      restartPolicy: Never
</code></pre>
<p>And please note this:</p>
<blockquote>
<p>Note that the TTL period, e.g. .spec.ttlSecondsAfterFinished field of Jobs, can be modified after the job is created or has finished. However, once the Job becomes eligible to be deleted (when the TTL has expired), the system won't guarantee that the Jobs will be kept, even if an update to extend the TTL returns a successful API response.</p>
</blockquote>
<p>For more information: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/#updating-ttl-seconds" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/#updating-ttl-seconds</a></p>
<p><strong>Solution 2</strong></p>
<p>As it mentioned above in the comment field, you can try to play with <code>kube-controller-manager</code> and increase the number of TTL-after-finished controller workers that are allowed to sync concurrently by using the following <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="nofollow noreferrer">flag option</a>:</p>
<p><code>kube-controller-manager --concurrent-ttl-after-finished-syncs int32     Default: 5</code></p>
