<p>I want to test the queuing and preemption features of Kubernetes (v1.21.0). I run Kubernetes using Minikube with a pod limit of 10. I have a script that creates two priority classes: 'low-priority' and 'high-priority'.</p>
<p><a href="https://i.stack.imgur.com/P5dpe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P5dpe.png" alt="Priority Classes" /></a></p>
<p>I then have a script that creates 10 low priority jobs, waits 20 seconds, and then creates a high priority one. In this scenario, one of the low priority ones is correctly terminated so that the high priority job can be executed.</p>
<p><a href="https://i.stack.imgur.com/dGfBF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dGfBF.png" alt="Preemption Working" /></a></p>
<p>I then have another script that does the same thing, but in a namespace with resource quotas:</p>
<pre><code>kubectl create namespace limited-cpu
cat &lt;&lt;EOF | kubectl apply -n limited-cpu -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: limit-max-cpu
spec:
  hard:
    requests.cpu: &quot;1000m&quot;
EOF
</code></pre>
<p>In this scenario, the low priority jobs request 333m of cpu and the high priority one 500m. The expected behavior is for Kubernetes to run three low priority at the same time, then to stop two of them when the high priority one is submitted.</p>
<p>But it does not happen. Worst: even when the low priority jobs end, other low priority jobs are scheduled before the high priority one.</p>
<p><a href="https://i.stack.imgur.com/uQHY1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uQHY1.png" alt="Preemption and queuing not working with resource quotas" /></a></p>
<p>Here are the two jobs definitions:</p>
<pre><code>for i in $(seq -w 1 10) ;
do
    cat &lt;&lt;EOF | kubectl apply -n limited-cpu -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: low-priority-$i
spec:
  template:
    spec:
      containers:
      - name: low-priority-$i
        image: busybox
        command: [&quot;sleep&quot;,  &quot;60s&quot;]
        resources:
          requests:
            memory: &quot;64Mi&quot;
            cpu: &quot;333m&quot;
      restartPolicy: Never
      priorityClassName: &quot;low-priority&quot;
EOF
done

sleep 20s

cat &lt;&lt;EOF | kubectl apply -n limited-cpu -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: high-priority-1
spec:
  template:
    spec:
      containers:
      - name: high-priority-1
        image: busybox
        command: [&quot;sleep&quot;,  &quot;30s&quot;]
        resources:
          requests:
            memory: &quot;128Mi&quot;
            cpu: &quot;500m&quot;
      restartPolicy: Never
      priorityClassName: &quot;high-priority&quot;
EOF

</code></pre>
<p>Even the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#preemption" rel="nofollow noreferrer">Kubernetes Documentation</a> agrees that it should be working.</p>
<p>EDIT:
Here are the Priority Classes definitions:</p>
<pre><code>cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
description: Low-priority Priority Class
kind: PriorityClass
metadata:
  name: low-priority
value: 1000000
EOF

cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
description: Low-priority Priority Class
kind: PriorityClass
metadata:
  name: high-priority
value: 99999999
EOF
</code></pre>

Gaëtan
<p>Fair question, fair assumption.</p>
<p>I've run into a similar situation and was also disappointed to see that k8s does not evict low-priority pods in favor of high-priority ones.</p>
<p>A couple of consultations with k8s experts revealed that indeed, when framed into namespace quotas, k8s is not expected to be as aggressive as in pure &quot;pods x nodes&quot; setup.</p>
<p>The official documentation you point to also seems to describe everything only in the context of &quot;pods x nodes&quot;.</p>
