<p>This morning I learned about the (unfortunate) default in kubernetes of all previously run cronjobs' jobs instances being retained in the cluster.  Mea culpa for not reading that detail in the documentation.  I also notice that deleting jobs (<code>kubectl delete job [&lt;foo&gt; or --all]</code>) takes quite a long time.  Further, I noticed that even a reasonably provisioned kubernetes cluster with three large nodes appears to fail (get timeouts of all sorts when trying to use kubectl) when there are just ~750 such old jobs in the system (plus some other active containers that otherwise had not entailed heavy load) [Correction: there were also ~7k pods associated with those old jobs that were also retained :-o].  (I did learn about <a href="https://stackoverflow.com/a/43115763/379037">the configuration settings</a> to limit/avoid storing old jobs from cronjobs, so this won't be a problem [for me] in the future.)</p>

<p>So, since I couldn't find documentation for kubernetes about this, my (related) questions are: </p>

<ol>
<li>what exactly is stored when kubernetes retains old jobs? (Presumably it's the associated pod's logs and some metadata, but this doesn't explain why they seemed to place such a load on the cluster.)</li>
<li>is there a way to see the resources (disk only, I assume, but maybe
there is some other resource) that individual or collective old jobs
are using?</li>
<li>why does deleting a kubernetes job take on the order of a minute?</li>
</ol>

<p>I don't know if k8s provides that kinda details of what job is consuming how much disk space but here is something you can try.</p>

<p>Try to find the pods associated with the job:</p>

<pre><code>kubectl get pods --selector=job-name=&lt;job name&gt; --output=jsonpath={.items..metadata.name}
</code></pre>

<p>Once you know the pod then find the docker container associated with it:</p>

<pre><code>kubectl describe pod &lt;pod name&gt;
</code></pre>

<p>In the above output look for <code>Node</code> &amp; <code>Container ID</code>. Now go on that node and in that node goto path <code>/var/lib/docker/containers/&lt;container id found above&gt;</code> here you can do some investigation to find out what is wrong.</p>
