<p>I have a Kubernetes cluster with auto-provisioning enabled on GKE.</p>

<pre><code>gcloud beta container clusters create "some-name" --zone "us-central1-a" \
  --no-enable-basic-auth --cluster-version "1.13.11-gke.14" \
  --machine-type "n1-standard-1" --image-type "COS" \
  --disk-type "pd-standard" --disk-size "100" \
  --metadata disable-legacy-endpoints=true \
  --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
  --num-nodes "1" --enable-stackdriver-kubernetes --enable-ip-alias \
  --network "projects/default-project/global/networks/default" \
  --subnetwork "projects/default-project/regions/us-central1/subnetworks/default" \
  --default-max-pods-per-node "110" \
  --enable-autoscaling --min-nodes "0" --max-nodes "8" \
  --addons HorizontalPodAutoscaling,KubernetesDashboard \
  --enable-autoupgrade --enable-autorepair \
  --enable-autoprovisioning --min-cpu 1 --max-cpu 40 --min-memory 1 --max-memory 64
</code></pre>

<p>I ran a deployment which wouldn't fit on the existing node (which has 1 CPU).</p>

<pre><code>kubectl run say-lol --image ubuntu:18.04 --requests cpu=4 -- bash -c 'echo lolol &amp;&amp; sleep 30'
</code></pre>

<p>The auto-provisioner correctly detected that a new node pool was needed, and it created a new cluster and started running the new deployment. <strong>However, it was not able to delete it after it was no longer needed.</strong></p>

<pre><code>kubectl delete deployment say-lol
</code></pre>

<p>After all pods are gone, the new cluster has been sitting idle for more than 20 hours.</p>

<pre><code>$ kubectl get nodes
NAME                                                  STATUS   ROLES    AGE   VERSION
gke-some-name-default-pool-5003d6ff-pd1p        Ready    &lt;none&gt;   21h   v1.13.11-gke.14
gke-some-name-nap-n1-highcpu-8--585d94be-vbxw   Ready    &lt;none&gt;   21h   v1.13.11-gke.14

$ kubectl get deployments
No resources found in default namespace.

$ kubectl get events
No resources found in default namespace.
</code></pre>

<p>Why isn't it cleaning up the expensive node pool?</p>

Andy Carlson
<p>In addition to the accepted answer, there is an approach using <strong>taints</strong>. If the un-schedulable pod has any tolerations, the auto-provisioner will create a nodes in the new node-pool with matching taints (<a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning#workload_separation" rel="nofollow noreferrer">see docs</a>). Because the new nodes are tainted, other pods will not run on them and prevent them from scaling down. I find this approach simpler and easier to understand than the PDB approach.</p>
