<p>I have the following pods on my kubernetes (1.18.3) cluster:</p>

<pre><code>NAME      READY   STATUS    RESTARTS   AGE
pod1      1/1     Running   0          14m
pod2      1/1     Running   0          14m
pod3      0/1     Pending   0          14m
pod4      0/1     Pending   0          14m
</code></pre>

<p>pod3 and pod4 cannot start because the node has capacity for 2 pods only. When pod1 finishes and quits, then the scheduler picks either pod3 or pod4 and starts it. So far so good.</p>

<p>However, I also have a high priority pod (hpod) that I'd like to start before pod3 or pod4 when either of the running pods finishes and quits.</p>

<p>So I created a priorityclass can be found in the kubernetes docs:</p>

<pre><code>kind: PriorityClass
metadata:
  name: high-priority-no-preemption
value: 1000000
preemptionPolicy: Never
globalDefault: false
description: "This priority class should be used for XYZ service pods only."
</code></pre>

<p>I've created the following pod yaml:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: hpod
  labels:
    app: hpod
spec:
  containers:
  - name: hpod
    image: ...
    resources:
      requests:
        cpu: "500m"
        memory: "500Mi"
      limits:
        cpu: "500m"
        memory: "500Mi"
  priorityClassName: high-priority-no-preemption
</code></pre>

<p>Now the problem is that when I start the high prio pod with kubectl apply -f hpod.yaml, then the scheduler terminates a running pod to allow the high priority pod to start despite I've set 'preemptionPolicy: Never'.</p>

<p>The expected behaviour would be to postpone starting hpod until a currently running pod finishes. And when it does, then let hpod start before pod3 or pod4.</p>

<p>What am I doing wrong?</p>

<p><strong>Prerequisites:</strong></p>
<p>This solution was tested on Kubernetes <code>v1.18.3</code>, docker 19.03 and Ubuntu 18.
Also text editor is required (i.e. <code>sudo apt-get install vim</code>).</p>
<p>In Kubernetes documentation under <a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#how-to-disable-preemption" rel="nofollow noreferrer">How to disable preemption</a> you can find <strong>Note</strong>:</p>
<blockquote>
<p>Note: In Kubernetes <code>1.15</code> and later, if the feature <code>NonPreemptingPriority</code> is <code>enabled</code>, <code>PriorityClasses</code> have the option to set <code>preemptionPolicy: Never</code>. This will prevent pods of that <code>PriorityClass</code> from preempting other pods.</p>
</blockquote>
<p>Also under <a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#non-preempting-priority-class" rel="nofollow noreferrer">Non-preempting PriorityClass</a> you have information:</p>
<blockquote>
<p>The use of the PreemptionPolicy field requires the <code>NonPreemptingPriority</code> feature gate to be <code>enabled</code>.</p>
</blockquote>
<p>Later if you will check thoses <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" rel="nofollow noreferrer">Feature Gates</a> info, you will find that <code>NonPreemptingPriority</code> is <code>false</code>, so as default it's disabled.</p>
<p>Output with your current configuration:</p>
<pre><code>$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
nginx-normal     1/1     Running   0          32s
nginx-normal-2   1/1     Running   0          32s
$ kubectl apply -f prio.yaml
pod/nginx-priority created$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
nginx-normal-2   1/1     Running   0          48s
nginx-priority   1/1     Running   0          8s
</code></pre>
<p>To enable <code>preemptionPolicy: Never</code> you need to apply <code>--feature-gates=NonPreemptingPriority=true</code> to 3 files:</p>
<blockquote>
<p>/etc/kubernetes/manifests/kube-apiserver.yaml</p>
<p>/etc/kubernetes/manifests/kube-controller-manager.yaml</p>
<p>/etc/kubernetes/manifests/kube-scheduler.yaml</p>
</blockquote>
<p>To check if this <code>feature-gate</code> is enabled you can check by using commands:</p>
<pre><code>ps aux | grep apiserver | grep feature-gates
ps aux | grep scheduler | grep feature-gates
ps aux | grep controller-manager | grep feature-gates
</code></pre>
<p>For quite detailed information, why you have to edit thoses files please check <a href="https://github.com/kubernetes/kubernetes/issues/86680" rel="nofollow noreferrer">this Github thread</a>.</p>
<pre><code>$ sudo su
# cd /etc/kubernetes/manifests/
# ls
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
</code></pre>
<p>Use your text editor to add feature gate to those files</p>
<pre><code># vi kube-apiserver.yaml
</code></pre>
<p>and add <code>- --feature-gates=NonPreemptingPriority=true</code> under <code>spec.containers.command</code> like in example bellow:</p>
<pre><code>spec:
  containers:
  - command:
    - kube-apiserver
    - --feature-gates=NonPreemptingPriority=true
    - --advertise-address=10.154.0.31
</code></pre>
<p>And do the same with 2 other files. After that you can check if this flags were applied.</p>
<pre><code>$ ps aux | grep apiserver | grep feature-gates
root     26713 10.4  5.2 565416 402252 ?       Ssl  14:50   0:17 kube-apiserver --feature-gates=NonPreemptingPriority=true --advertise-address=10.154.0.31 
</code></pre>
<p>Now you have redeploy your <code>PriorityClass</code>.</p>
<pre><code>$ kubectl get priorityclass
NAME                          VALUE        GLOBAL-DEFAULT   AGE
high-priority-no-preemption   1000000      false            12m
system-cluster-critical       2000000000   false            23m
system-node-critical          2000001000   false            23m
$ kubectl delete priorityclass high-priority-no-preemption
priorityclass.scheduling.k8s.io &quot;high-priority-no-preemption&quot; deleted
$ kubectl apply -f class.yaml 
priorityclass.scheduling.k8s.io/high-priority-no-preemption created
</code></pre>
<p>Last step is to deploy pod with this <code>PriorityClass</code>.</p>
<p><strong>TEST</strong></p>
<pre><code>$ kubectl get po
NAME             READY   STATUS    RESTARTS   AGE
nginx-normal     1/1     Running   0          4m4s
nginx-normal-2   1/1     Running   0          18m
$ kubectl apply -f prio.yaml 
pod/nginx-priority created
$ kubectl get po
NAME             READY   STATUS    RESTARTS   AGE
nginx-normal     1/1     Running   0          5m17s
nginx-normal-2   1/1     Running   0          20m
nginx-priority   0/1     Pending   0          67s
$ kubectl delete po nginx-normal-2
pod &quot;nginx-normal-2&quot; deleted
$ kubectl get po
NAME             READY   STATUS    RESTARTS   AGE
nginx-normal     1/1     Running   0          5m55s
nginx-priority   1/1     Running   0          105s
</code></pre>
