<p>On my kubernetes nodes there are</p>
<ol>
<li>prioritized pods</li>
<li>dispensable pods</li>
</ol>
<p>Therefore I would like to have QoS class of <code>Guaranteed</code> for the prioritized pods.
To achieve a <code>Guaranteed</code> class the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="nofollow noreferrer">cpu/memory requests/limits must meet some conditions</a>. Therefore:</p>
<blockquote>
<p>For every Container in the Pod, the CPU limit must equal the CPU
request</p>
</blockquote>
<p>But I would like to set a higher CPU limit than request, so that the prioritized pods can use every free CPU resources which are available.</p>
<p>Simple example: A Node with 4 cores has:</p>
<ul>
<li>1 prioritized pod with 2000 CPU request and 3900 CPU limit</li>
<li>3 dispensable pods with each 500 CPU request and limit.</li>
</ul>
<p>If the prioritized pod would have 2000 CPU request and limit 2 Cores are wasted because the dispensable pods don't use CPU most of the time.</p>
<p>If the prioritized pod would have 3900 CPU request and limit, I would need an extra node for the dispensable pods.</p>
<p><strong>Questions</strong></p>
<p>Is it possible to set explicitly the <code>Guaranteed</code> class to a pod even with difference CPU request and limit?</p>
<p>If it's not possible: Why is there no way to explicitly set the QoS class?</p>
<p><strong>Remarks</strong></p>
<p>There's an <a href="https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/" rel="nofollow noreferrer">system-cluster-critical</a> option. But I think this should only be used for critical k8s add-on pods but not for critical applications.</p>

<blockquote>
<p>Is it possible to set explicitly the <code>Guaranteed</code> class to a pod even with difference CPU request and limit?</p>
</blockquote>
<p><strong>Yes, however you will need to use an additional plugin: <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/kep/9-capacity-scheduling" rel="nofollow noreferrer">capacity-scheduling</a> used with <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/#resource-quota-per-priorityclass" rel="nofollow noreferrer"><code>PriorityClass</code></a>:</strong></p>
<blockquote>
<p>There is increasing demand to use Kubernetes to manage batch workloads (ML/DL). In those cases, one challenge is to improve cluster utilization while ensuring that each user has a reasonable amount of resources. The problem can be partially addressed by the Kubernetes <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">ResourceQuota</a>. The native Kubernetes ResourceQuota API can be used to specify the maximum overall resource allocation per namespace. The quota enforcement is done through an admission check. A quota consumer (e.g., a Pod) cannot be created if the aggregated resource allocation exceeds the quota limit. In other words, the overall resource usage is aggregated based on Pod's spec (i.e., cpu/mem requests) when it's created. The Kubernetes quota design has the limitation: the quota resource usage is aggregated based on the resource configurations (e.g., Pod cpu/mem requests specified in the Pod spec). Although this mechanism can guarantee that the actual resource consumption will never exceed the ResourceQuota limit, it might lead to low resource utilization as some pods may have claimed the resources but failed to be scheduled. For instance, actual resource consumption may be much smaller than the limit.</p>
</blockquote>
<hr />
<blockquote>
<p>Pods can be created at a specific  <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#pod-priority" rel="nofollow noreferrer">priority</a>. You can control a pod's consumption of system resources based on a pod's priority, by using the  <code>scopeSelector</code>  field in the quota spec.</p>
<p>A quota is matched and consumed only if  <code>scopeSelector</code>  in the quota spec selects the pod.</p>
<p>When quota is scoped for priority class using  <code>scopeSelector</code>  field, quota object is restricted to track only following resources:</p>
<ul>
<li><code>pods</code></li>
<li><code>cpu</code></li>
<li><code>memory</code></li>
<li><code>ephemeral-storage</code></li>
<li><code>limits.cpu</code></li>
<li><code>limits.memory</code></li>
<li><code>limits.ephemeral-storage</code></li>
<li><code>requests.cpu</code></li>
<li><code>requests.memory</code></li>
<li><code>requests.ephemeral-storage</code></li>
</ul>
</blockquote>
<p>This plugin supports also <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/kep/9-capacity-scheduling#additional-preemption-details" rel="nofollow noreferrer">preemption</a> (example for Elastic):</p>
<blockquote>
<p>Preemption happens when a pod is unschedulable, i.e., failed in PreFilter or Filter phases.</p>
<p>In particular for capacity scheduling, the failure reasons could be:</p>
<ul>
<li>Prefilter Stage</li>
<li>sum(allocated res of pods in the same elasticquota) + pod.request &gt; elasticquota.spec.max</li>
<li>sum(allocated res of pods in the same elasticquota) + pod.request &gt; sum(elasticquota.spec.min)</li>
</ul>
<p>So the preemption logic will attempt to make the pod schedulable, with a cost of preempting other running pods.</p>
</blockquote>
<p>Examples of yaml files and usage can be found <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/kep/9-capacity-scheduling#design-details" rel="nofollow noreferrer">in the plugin description</a>.</p>
