<p>I have multi-environment k8s cluster (<strong>EKS</strong>) and I'm trying to setup accurate values for ResourceQuotas. </p>

<p>One interesting thing that I've noticed is that specified request/limit for CPU/memory stay <strong>"occupied"</strong> in k8s cluster when job is completed successfully and effectively pod releases cpu/memory resources that it is using. </p>

<p>Since I expect that there would be a lot of jobs executed on the environment this caused a problem for me. Of course, I've added support for running cleanup cronjob for the successfully executed jobs but that is just one part of the solution.</p>

<p>I'm aware of the <strong>TTL feature on k8s</strong>: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#ttl-mechanism-for-finished-jobs" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#ttl-mechanism-for-finished-jobs</a> that is still in alpha state and as such not available on the EKS k8s cluster.</p>

<p>I would expect that both request/limits specified on that specific pod (container/s) are "released" also but when looking at the k8s metrics on Grafana, I see that that is not true.</p>

<p>This is an example (green line marks current resource usage, yellow marks resource request while blue marks resource limit):
<a href="https://i.stack.imgur.com/5SfMZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5SfMZ.png" alt="enter image description here"></a></p>

<p>My question is:</p>

<ul>
<li>Is this expected behaviour?</li>
<li>If yes, what are the technical reasons why request/limits are not released as well after job (pod) execution is completed?</li>
</ul>

<p>I've done "load" test on my environment to test if requests/limits that are left assigned on the completed job (pod) will indeed have influence on the ResourceQuota that I've set. </p>

<p>This is how my ResourceQuota looks like:</p>

<pre><code>apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-quota
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 2Gi
    limits.cpu: "2"
    limits.memory: 3Gi
</code></pre>

<p>This is request/limit for cpu/memory that exists on each k8s job (to be precise on the container running in the Pod which is spinned up by Job):</p>

<pre><code>resources:
  limits:
    cpu: 250m
    memory: 250Mi
  requests:
    cpu: 100m
    memory: 100Mi
</code></pre>

<p>Results of testing:</p>

<ul>
<li>Currently running number of jobs: <strong>66</strong></li>
<li>Expected sum of CPU requests (if assumption from the question is correct) <strong>~= 6.6m</strong></li>
<li>Expected sum of Memory requests (if assumption from the question is correct) <strong>~= 6.6Mi</strong></li>
<li>Expected sum of CPU limits (if assumption from the question is correct) <strong>~= 16.5</strong></li>
<li>Expected sum of Memory limits (if assumption from the question is correct) <strong>~= 16.5</strong></li>
</ul>

<p>I've created Grafana graphs that show following:</p>

<p><strong>CPU usage/requests/limits for jobs in one namespace</strong></p>

<pre><code>sum(rate(container_cpu_usage_seconds_total{namespace="${namespace}", container="myjob"}[5m]))
sum(kube_pod_container_resource_requests_cpu_cores{namespace="${namespace}", container="myjob"})
sum(kube_pod_container_resource_limits_cpu_cores{namespace="${namespace}", container="myjob"})
</code></pre>

<p><strong>Memory usage/requests/limits for jobs in one namespace</strong></p>

<pre><code>sum(rate(container_memory_usage_bytes{namespace="${namespace}", container="myjob"}[5m]))
sum(kube_pod_container_resource_requests_memory_bytes{namespace="${namespace}", container="myjob"})
sum(kube_pod_container_resource_limits_memory_bytes{namespace="${namespace}", container="myjob"})
</code></pre>

<p>This is how graphs look like:
<a href="https://i.stack.imgur.com/MSa3M.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MSa3M.png" alt="enter image description here"></a></p>

<p>According to this graph, requests/limits get accumulated and go well beyond the ResourceQuota thresholds. However, I'm still able to run new jobs without a problem.</p>

<p>At this moment, I've started doubting in what metrics are showing and opted to check other part of the metrics. To be specific, I've used following set of metrics:</p>

<p><strong>CPU:</strong></p>

<pre><code>sum (rate(container_cpu_usage_seconds_total{namespace="$namespace"}[1m]))
kube_resourcequota{namespace="$namespace", resource="limits.cpu", type="hard"}
kube_resourcequota{namespace="$namespace", resource="requests.cpu", type="hard"}
kube_resourcequota{namespace="$namespace", resource="limits.cpu", type="used"}
kube_resourcequota{namespace="$namespace", resource="requests.cpu", type="used"}
</code></pre>

<p><strong>Memory:</strong></p>

<pre><code>sum (container_memory_usage_bytes{image!="",name=~"^k8s_.*", namespace="$namespace"})
kube_resourcequota{namespace="$namespace", resource="limits.memory", type="hard"}
kube_resourcequota{namespace="$namespace", resource="requests.memory", type="hard"}
kube_resourcequota{namespace="$namespace", resource="limits.memory", type="used"}
kube_resourcequota{namespace="$namespace", resource="requests.memory", type="used"}
</code></pre>

<p>This is how graph looks like:
<a href="https://i.stack.imgur.com/jp7BQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jp7BQ.png" alt="enter image description here"></a></p>

<hr>

<p><strong>Conclusion:</strong></p>

<p>From this screenshot, it is clear that, once load test completes and jobs go into the complete state, even though pods are still around <strong>(with READY: 0/1 and STATUS: Completed)</strong>, cpu/memory request/limits are released and no longer represent constraint that needs to be calculated into the ResourceQuota threshold. 
This can be seen by observing following data on the graph:</p>

<pre><code>CPU allocated requests
CPU allocated limits
Memory allocated requests
Memory allocated limits
</code></pre>

<p>all of which increase at the point of time when load happens on the system and new jobs are created but goes back into the previous state as soon as jobs are completed (even though they are not deleted from the environment)</p>

<p><strong>In other words, resource usage/limits/request for both cpu/memory are taken into the account only while job (and its corresponding pod) is in RUNNING state</strong> </p>
