<p>I have autoscaling enabled on Google Kubernetes Cluster and one of the pods I can see the usage is much lower</p>

<p><a href="https://i.stack.imgur.com/jHLLT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jHLLT.png" alt="enter image description here"></a></p>

<p>I have a total of 6 nodes and I expect at least this node to be terminated. I have gone through the following:
<a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node</a></p>

<p>I have added this annotation to all my pods </p>

<pre><code>cluster-autoscaler.kubernetes.io/safe-to-evict: true
</code></pre>

<p>However, the cluster autoscaler scales up correctly, but doesn't scale down as I expect it to.</p>

<p>I have the following logs</p>

<pre><code>$ kubectl  logs kube-dns-autoscaler-76fcd5f658-mf85c -n kube-system

autoscaler/pkg/autoscaler/k8sclient/k8sclient.go:90: Failed to list *v1.Node: Get https://10.55.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.55.240.1:443: getsockopt: connection refused
E0628 20:34:36.187949       1 reflector.go:190] github.com/kubernetes-incubator/cluster-proportional-autoscaler/pkg/autoscaler/k8sclient/k8sclient.go:90: Failed to list *v1.Node: Get https://10.55.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.55.240.1:443: getsockopt: connection refused
E0628 20:34:47.191061       1 reflector.go:190] github.com/kubernetes-incubator/cluster-proportional-autoscaler/pkg/autoscaler/k8sclient/k8sclient.go:90: Failed to list *v1.Node: Get https://10.55.240.1:443/api/v1/nodes?resourceVersion=0: net/http: TLS handshake timeout
I0628 20:35:10.248636       1 autoscaler_server.go:133] ConfigMap not found: Get https://10.55.240.1:443/api/v1/namespaces/kube-system/configmaps/kube-dns-autoscaler: net/http: TLS handshake timeout, will create one with default params
E0628 20:35:17.356197       1 autoscaler_server.go:95] Error syncing configMap with apiserver: configmaps "kube-dns-autoscaler" already exists
E0628 20:35:18.191979       1 reflector.go:190] github.com/kubernetes-incubator/cluster-proportional-autoscaler/pkg/autoscaler/k8sclient/k8sclient.go:90: Failed to list *v1.Node: Get https://10.55.240.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.55.240.1:443: i/o timeout
</code></pre>

<p>I am not sure the above are the relevant logs, what is the correct way to debug this issue?</p>

<p>My pods have got local storage. I have been trying to debug this issue using </p>

<pre><code>kubectl drain  gke-mynode-d57ded4e-k8tt

error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): fluentd-gcp-v3.1.1-qzdzs, prometheus-to-sd-snqtn; pods with local storage (use --delete-local-data to override): mydocs-585879b4d5-g9flr, istio-ingressgateway-9b889644-v8bgq, mydocs-585879b4d5-7lmzk
</code></pre>

<p>I think it's safe to ignore <code>daemonsets</code> as CA should be ok to evict it, however I am not sure how to make the CA understand that mydocs is ok to be evicted and move to another node after adding the annotation</p>

<h1>EDIT</h1>

<p>The min and the max nodes have been set correctly as seen on the GCP console
<a href="https://i.stack.imgur.com/SNZUa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SNZUa.png" alt="enter image description here"></a></p>

<p>The <code>kubectl logs</code> command is for the DNS autoscaler, not the cluster autoscaler. It will give you information on the number of kube-dns replicas in the cluster, not the number of nodes or scaling decisions.</p>

<p>From the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">cluster autoscaler FAQ</a> (and taking into account what you wrote in your question):</p>

<blockquote>
  <p>Kube-system pods that:</p>
  
  <ul>
  <li>are not run on the node by default</li>
  <li>Pods with local storage</li>
  </ul>
</blockquote>

<p>And additionally, restrictive <code>Pod Disruption Budgets</code>. However since is not stated in the question, I'll assume you haven't set any.</p>

<p>Although you have pods with local storage, you added the annotation to make them safe to evict so that leaves the system pods not run by default in the nodes.</p>

<p>Since system pods in GKE are annotated with the <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/addon-manager" rel="nofollow noreferrer">reconciliation loop</a>, you can't add this directive to them, which might be preventing their eviction.</p>

<p>In this scenario, you may consider using a <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-my-cluster-to-just-1-node" rel="nofollow noreferrer"><code>Pod Disruption Budget</code> configured to allow the autoscaler to evict them</a>.</p>

<p>This <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-to-set-pdbs-to-enable-ca-to-move-kube-system-pods" rel="nofollow noreferrer"><code>Pod Disruption Budget</code></a> can include DNS and logging pods that aren't run by default in the nodes.</p>

<p>Unfortunately, GKE is a managed option so there isn't much to apply from the autoscaler FAQ. However, if you want to go further, you might as well consider a <a href="https://en.wikipedia.org/wiki/Bin_packing_problem" rel="nofollow noreferrer">pod binpacking strategy</a> using <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity" rel="nofollow noreferrer">Affinity and anti-affinity</a>, <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">Taints and tolerations</a> and <a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits" rel="nofollow noreferrer">requests and limits</a> to fit them properly, making the downscaling easier whenever possible.</p>

<p>Finally, on GKE you can use the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-check-what-is-going-on-in-ca-" rel="nofollow noreferrer"><code>cluster-autoscaler-status</code> ConfigMap</a> to check what decisions the autoscaler is making.</p>
