<p>Creating jobs like the following for example:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: batch/v1
kind: Job
metadata:
  name: test-job-sebas
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: [&quot;perl&quot;,  &quot;-Mbignum=bpi&quot;, &quot;-wle&quot;, &quot;print bpi(2000)&quot;]
      restartPolicy: Never
  backoffLimit: 4
</code></pre>
<p>Results in the job resource being created but no pod or event is observed. Pod statuses are as follow:</p>
<p><code>Pods Statuses:  1 Running / 0 Succeeded / 0 Failed</code></p>
<p>And the only event visible is the notification of a succesful pod being created. The problem is that the message appears only after 30 minutes of complete apparent silence.</p>
<p><code>Normal  SuccessfulCreate  21m   job-controller  Created pod: test-job-sebas-882bh</code></p>
<p>From the time we can observe the <code>kube-apiserver</code> log allowing the &quot;create&quot; verb for the Job resource, we are unable to spot any other log in any of the other pods (controllers/schedulers/apiserver) that have the text &quot;test-job-sebas&quot;, until ~30 minutes later where the <code>kube-controller-manager</code> logs the following.</p>
<pre><code>Event occurred&quot; object=&quot;test-namespace/job-test-01&quot; kind=&quot;Job&quot; apiVersion=&quot;batch/v1&quot; type=&quot;Normal&quot; reason=&quot;SuccessfulCreate&quot; message=&quot;Created pod: test-job-sebas-882bh&quot;
</code></pre>
<p>This happens with any Job in this cluster, no matter which namespace or the nature of the Job, if it comes from a CronJob or if it is explicitly created like the one provided in the example here.</p>
<p>Looking at the code does not throw any obvious suspect for us that points to what could be happening: <a href="https://github.com/kubernetes/kubernetes/blob/b5b0cc8bb88fb678c9b065c8da4f4c06a155a628/pkg/controller/job/job_controller.go" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/blob/b5b0cc8bb88fb678c9b065c8da4f4c06a155a628/pkg/controller/job/job_controller.go</a></p>
<p>edit: We currently have ~15.000 jobs in the cluster where it seems that most of them are active, from only one namespace. This would lead us to think that we are hitting some limit or making some sort of saturation...but we can't confirm this by any of the visible data.</p>

carrotcakeslayer
<p>This sounds very similar to what I encountered when we had a misbehaving webhook.</p>
<p>If you have a massive number of jobs all showing as active, but no pods appearing, or pods taking a long time to appear, then that's a sign of an admission webhook interfering with the pod creation. If it's a cronjob affected, you will get a &quot;snowball&quot; effect:</p>
<p>Writeup: <a href="https://blenderfox.com/2020/08/07/the-snowball-effect-in-kubernetes/" rel="nofollow noreferrer">https://blenderfox.com/2020/08/07/the-snowball-effect-in-kubernetes/</a></p>
<p>Kubernetes Issue: <a href="https://github.com/kubernetes/kubernetes/issues/93783" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/93783</a></p>
<p>As for fixing your issue, you need to find out what is interfering with the creation (in our case, we had an up9 webhook misbehaving. Disabling that allowed the creation of the pods)</p>
