<p>I'm setting up a GKE cluster on Google Kubernetes Engine to run some heavy jobs. I have a render-pool of big machines that I want to autoscale from 0 to N (using the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler" rel="nofollow noreferrer">cluster autoscaler</a>). My default-pool is a cheap g1-small to run the system pods (those never go away so the default pool can't autoscale to 0, too bad).</p>

<p>My problem is that the render-pool doesn't want to scale down to 0. It has some system pods running on it; are those the problem? The default pool has plenty of resources to run all of them as far as I can tell. I've read the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#when-does-cluster-autoscaler-change-the-size-of-a-cluster" rel="nofollow noreferrer">autoscaler FAQ</a>, and it looks like it should delete my node after 10 min of inactivity. I've waited an hour though.</p>

<p>I created the render pool like this:</p>

<pre><code>gcloud container node-pools create render-pool-1 --cluster=test-zero-cluster-2 \
 --disk-size=60 --machine-type=n2-standard-8 --image-type=COS \
 --disk-type=pd-standard --preemptible --num-nodes=1 --max-nodes=3 --min-nodes=0 \
 --enable-autoscaling
</code></pre>

<p>The cluster-autoscaler-status configmap says <code>ScaleDown: NoCandidates</code> and it is probing the pool frequently, as it should.</p>

<p>What am I doing wrong, and how do I debug it? Can I see why the autoscaler doesn't think it can delete the node?</p>

<p>As pointed out in the comments, <a href="https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.2.2/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">some pods, under specific circumstances will prevent the CA from downscaling</a>.</p>

<p>In GKE, you have logging pods (fluentd), kube-dns, monitoring, etc., all considered system pods. This means that any node where they're scheduled, will <strong>not</strong> be a candidate for downscaling.</p>

<p>Considering this, it all boils down to creating an scenario where all the <a href="https://github.com/kubernetes/autoscaler/blob/cluster-autoscaler-1.2.2/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">previous conditions</a> for downscaling are met.</p>

<p>Since you only want to scale down an specific node-pool, I'd use <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">Taints and tolerations</a> to keep system pods in the default pool.</p>

<p>For GKE specifically, you can pick each app by their <code>k8s-app</code> label, for instance:</p>

<p><code>$ kubectl taint nodes GPU-NODE k8s-app=heapster:NoSchedule</code></p>

<p>This will prevent the tainted nodes from scheduling <em>Heapster</em>.</p>

<p>Not recommended but, you can go broader and try to get all the GKE system pods using <code>kubernetes.io/cluster-service</code> instead:</p>

<p><code>$ kubectl taint nodes GPU-NODE kubernetes.io/cluster-service=true:NoSchedule</code></p>

<p>Just be careful as the scope of this label is broader and you'll have to keep track of oncoming changes, as this label is possibily <a href="https://github.com/kubernetes/kubernetes/issues/72757#issuecomment-490664478" rel="nofollow noreferrer">going to be deprecated someday</a>.</p>

<p>Another thing that you might want to consider is using <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/" rel="nofollow noreferrer">Pod Disruption Budgets</a>. This might be more effective in <strong>stateless</strong> workloads, but setting it very tight is likely to cause inestability.</p>

<p>The idea of a PDB is to tell GKE what's the very minimal amount of pods that can be run at any given time, allowing the CA to evict them. It can be applied to system pods like below:</p>

<pre><code>apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: dns-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
</code></pre>

<p>This tell GKE that, although there's usually 3 replicas of <em>kube-dns</em>, the application <em>might</em> be able to take 2 disruptions and sustain temporarily with only 1 replica, allowing the CA to evict these pods and reschedule them in other nodes.</p>

<p>As you probably noticed, this will put stress on DNS resolution in the cluster (in this particular example), so be careful.</p>

<p>Finally and regarding how to debug the CA. For now, consider that GKE is a managed version of Kubernetes where you don't really have direct access to tweak some features (for better or worse). You cannot set flags in the CA and access to logs could be through GCP support. The idea is to protect the workloads running in the cluster rather than to be cost-wise.</p>

<p>Downscaling in GKE is more about using different features in Kubernetes together until the CA conditions for downscaling are met.</p>
