<p>GKE cluster is configured with cluster/node auto provisioning. </p>

<p>I have created a default node-pool on which system specific pods can be run. Whenever pods with GPU are requested, GKE automatically creates new GPU enabled node pool which is fine.</p>

<p>But, whenever I delete such pods, GKE doesn't downscale newly create node pool to zero instances. Instead, one instance keeps running. If no GPU requested, node pool supposed to go to minimum size i.e. zero.</p>

<p>NOTE:</p>

<ul>
<li>For GPU drivers, a Daemonset has been created under 'kube-system' namespace, Pods for this Daemonsets run on each GPU enabled node.</li>
</ul>

<p>I edited this Daemonset and also added label '<em>"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"</em>
' to pods.</p>

<p>Can someone help how to downscale newly create node pool to zero nodes?</p>

<p>UPDATE:</p>

<p>Pods that are running on new nodes are:</p>

<p>fluentd-gcp  (From DaemonSet)</p>

<p>kube-proxy </p>

<p>nvidia-gpu-device-plugin (From DaemonSet)</p>

<p>Aren't these pods should get evicted ?</p>

<p>GKE by default keeps an extra node resource for quick pod scheduling. This is default behavior controlled by auto scaling policy.</p>

<p>This behavior can be changed by setting policy to 'optimize-utilization'. </p>

<p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler</a></p>
