<p>I have a Kubernetes cluster on GKE that's managed by Helm, and recently deploys started failing because pods would not leave the <code>Pending</code> state. Inspecting the pending pods I see:</p>

<pre><code>Events:
  Type     Reason            Age                From                                                Message
  ----     ------            ----               ----                                                -------
  Normal   Scheduled         50m                default-scheduler                                   Successfully assigned default/my-pod-6cbfb94cb-4pzl9 to gke-staging-pool-43f2e11c-nzjz
  Warning  FailedValidation  50m (x6 over 50m)  kubelet, gke-staging-pool-43f2e11c-nzjz  Error validating pod my-pod-6cbfb94cb-4pzl9_default(8e4dab93-75a7-11e9-80e1-42010a960181) from api, ignoring: spec.priority: Forbidden: Pod priority is disabled by feature-gate
</code></pre>

<p>And specifically, this warning seems relevant: <code>Error validating pod my-pod-6cbfb94cb-4pzl9_default(8e4dab93-75a7-11e9-80e1-42010a960181) from api, ignoring: spec.priority: Forbidden: Pod priority is disabled by feature-gate</code></p>

<p>Comparing the new, <code>Pending</code>, pods with currently running pods, the only difference I see (apart from timestamps, etc) is:</p>

<pre><code>$ kubectl get pod my-pod-6cbfb94cb-4pzl9 -o yaml &gt; /tmp/pending-pod.yaml
$ kubectl get pod my-pod-7958cc964-64wsd -o yaml &gt; /tmp/running-pod.yaml
$ diff /tmp/running-pod.yaml /tmp/pending-pod.yaml
…
@@ -58,7 +58,8 @@
       name: default-token-wnhwl
       readOnly: true
   dnsPolicy: ClusterFirst
-  nodeName: gke-staging-pool-43f2e11c-r4f9
+  nodeName: gke-staging-pool-43f2e11c-nzjz
+  priority: 0 // &lt;-- notice that the `priority: 0` field is added
   restartPolicy: Always
   schedulerName: default-scheduler
…
</code></pre>

<p>It appears that this started happening some time between May 1st, 2019, and May 6th 2019.</p>

<p>The cluster I've used as an example here is a staging cluster, but I've noticed the same behavior on two similarly configured production clusters, which leads me to believe there was a change on the Google Kube side.</p>

<p>Pods are deployed by Helm through <code>cloudbuild.yaml</code>, and nothing has changed in that setup (either version of Helm, or cloudbuild file) between May 1st and May 6th when the regression seems to have been introduced.</p>

<p>Helm version:</p>

<pre><code>Client: &amp;version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &amp;version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
</code></pre>

<p>If you see the docs for <a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/" rel="nofollow noreferrer">Pod priority and Preemption</a> the feature is <code>alpha</code> in <code>&lt;= 1.10</code> (disabled by default, enabled by a feature gate which GKE doesn't do on the control plane, afaik) then it became <code>beta</code> in <code>&gt;= 1.11</code> (enabled by default). </p>

<p>Could be either or combination:</p>

<ol>
<li><p>You have a GKE control plane <code>&gt;= 1.11</code> and the node where your Pod is trying to start (being scheduled by the kube-scheduler) is running a kubelet <code>&lt;= 1.10</code>. Somebody could have upgraded the control plane without upgrading the nodes (or instance group)</p></li>
<li><p>Someone upgraded the control plane to 1.11 or later and the priority admission controller is enabled by default and it's preventing Pods with the <code>spec.priority</code> field set from starting (or restarting). If you see the <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#pod-v1-core" rel="nofollow noreferrer">API docs</a> (priority field) it says that when the priority admission controller is enabled you can't set that field and that field can only be set by a <a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass" rel="nofollow noreferrer">PriorityClass/PriorityClassName</a>.</p></li>
</ol>
