<p>I have created a google Kubernetes engine with autoscale enabled with minimum and maximum nodes. A few days ago I deployed couple of servers on production which increased the nodes count as expected. but when I deleted those deployments I expect it to resize the nodes which are to scale down. I waited more than an hour but it still did not scale down.</p>
<ul>
<li>All my other pods are controlled by replica controller since I deployed with kind: deployment.</li>
<li>All my statefulset pods are using PVC as volume.</li>
</ul>
<p>I'm not sure what prevented the nodes to scale down so I manually scaled the nodes for now. Since I made the changes manually I can not get the autoscaler logs now.</p>
<p>Does anyone know what could be the issue here?</p>
<p>GKE version is 1.16.15-gke.4300</p>
<p>As mentioned in this link
<a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node</a></p>
<ul>
<li>I'm not using any local storage.</li>
<li>pods not having PodDisruptionBudget(don't know what is that)</li>
<li>Pods are created by deployments (helm charts)</li>
<li><strong>only thing is I don't have &quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;true&quot; this annotation. is this must?</strong></li>
</ul>

<p>I have tested <code>Cluster Autoscaler</code> on my GKE cluster. It work's bit differently than you expected.</p>
<h3>Backgorund</h3>
<p>You can enable autoscaling using command or enable it during creation like it's described <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler" rel="noreferrer">in this documentation</a>.</p>
<p>In <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler" rel="noreferrer">Cluster Autoscaler</a> documentation you can find various information like <code>Operation criteria</code>, <code>Limitations</code>, etc.</p>
<p>As I mentioned in comment section, <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="noreferrer">Cluster Autoscaler - Frequently Asked Questions</a> won't work if will encounter one of below situation:</p>
<blockquote>
<p>Pods with restrictive PodDisruptionBudget.</p>
<p>Kube-system pods that:</p>
<ul>
<li>are not run on the node by default, *</li>
<li>don't have a pod disruption budget set or their PDB is too restrictive (since CA 0.6).</li>
</ul>
<p>Pods that are not backed by a controller object (so not created by deployment, replica set, job, statefulset etc). *</p>
<p>Pods with local storage. *</p>
<p>Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, matching anti-affinity, etc)</p>
<p>Pods that have the following annotation set:
<code>&quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;</code></p>
</blockquote>
<p>For my tests I've used 6 nodes, with <code>autoscaling range 1-6</code> and <code>nginx</code> application with requests <code>cpu: 200m</code> and <code>memory: 128Mi</code>.</p>
<p>As OP mentioned that is not able to provide autoscaler logs, I will paste my logs from <code>Logs Explorer</code>. Description of how they can be achieved is in <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#viewing_events" rel="noreferrer">Viewing cluster autoscaler events</a> documentation.</p>
<p>In those logs you should search <code>noScaleDown</code> events. You will find there a few information, however the most important is:</p>
<pre><code>reason: {
parameters: [
0: &quot;kube-dns-66d6b7c877-hddgs&quot;
]
messageId: &quot;no.scale.down.node.pod.kube.system.unmovable&quot;
</code></pre>
<p>As it's described in <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#noscaledown-node-level-reasons" rel="noreferrer">NoScaleDown node-level reasons</a> for <code>&quot;no.scale.down.node.pod.kube.system.unmovable&quot;</code>:</p>
<blockquote>
<p>Pod is blocking scale down because it's a non-daemonset, non-mirrored, non-pdb-assigned kube-system pod. See the Kubernetes <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md" rel="noreferrer">Cluster Autoscaler FAQ</a> for more details.</p>
</blockquote>
<h3>Solution</h3>
<p>If you want to make <code>Cluster Autoscaler</code> work on <code>GKE</code>, you have to create <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#how-disruption-budgets-work" rel="noreferrer">Disruptions</a> with proper information, how to create it can be found in <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-to-set-pdbs-to-enable-ca-to-move-kube-system-pods" rel="noreferrer">How to set PDBs to enable CA to move kube-system pods?</a></p>
<pre><code>kubectl create poddisruptionbudget &lt;pdb name&gt; --namespace=kube-system --selector app=&lt;app name&gt; --max-unavailable 1
</code></pre>
<p>where you have to specify the correct <code>selector</code> and <code>--max-unavailable</code> or <code>--min-available</code> depends on your needs. For more details, please read <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget" rel="noreferrer">Specifying a PodDisruptionBudget</a> documentation.</p>
<h3>Tests</h3>
<pre><code>$ kubectl get deploy,nodes
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   16/16   16           16          66m

NAME                                            STATUS   ROLES    AGE     VERSION
node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Ready    &lt;none&gt;   11m     v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-2j4j   Ready    &lt;none&gt;   11m     v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-388n   Ready    &lt;none&gt;   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-5x35   Ready    &lt;none&gt;   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-pdfk   Ready    &lt;none&gt;   3h33m   v1.16.15-gke.6000
node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Ready    &lt;none&gt;   11m     v1.16.15-gke.6000
$ kubectl get pdb -A
NAMESPACE     NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kube-system   kubedns   1               N/A               1                     43m
</code></pre>
<p><strong>Scaledown deployment</strong></p>
<pre><code>$ kubectl scale deploy nginx-deployment --replicas=2
deployment.apps/nginx-deployment scaled
</code></pre>
<p>After a while (~10-15 minutes) in the event viewer you will find the <code>Decision</code> event and inside you will find information that the node was deleted.</p>
<pre><code>...
scaleDown: {
nodesToBeRemoved: [
0: {
node: {
mig: {
zone: &quot;europe-west2-c&quot;
nodepool: &quot;default-pool&quot;
name: &quot;gke-cluster-1-default-pool-6d42fa0a-grp&quot;
}
name: &quot;gke-cluster-1-default-pool-6d42fa0a-wqtm&quot;
</code></pre>
<p>Number of nodes decreased:</p>
<pre><code>$ kubectl get nodes
NAME                                       STATUS   ROLES    AGE     VERSION
gke-cluster-1-default-pool-6d42fa0a-2j4j   Ready    &lt;none&gt;   30m     v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-388n   Ready    &lt;none&gt;   3h51m   v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-5x35   Ready    &lt;none&gt;   3h51m   v1.16.15-gke.6000
gke-cluster-1-default-pool-6d42fa0a-pdfk   Ready    &lt;none&gt;   3h51m   v1.16.15-gke.6000
</code></pre>
<p>Another place where you can confirm it's scaling down is <code>kubectl get events  --sort-by='.metadata.creationTimestamp'</code></p>
<p><strong>Output:</strong></p>
<pre><code>5m16s       Normal    NodeNotReady                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm status is now: NodeNotReady
4m56s       Normal    NodeNotReady                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn status is now: NodeNotReady
4m          Normal    Deleting node gke-cluster-1-default-pool-6d42fa0a-wqtm because it does not exist in the cloud provider   node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm event: DeletingNode
3m55s       Normal    RemovingNode                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-wqtm   Node gke-cluster-1-default-pool-6d42fa0a-wqtm event: Removing Node gke-cluster-1-default-pool-6d42fa0a-wqtm from Controller
3m50s       Normal    Deleting node gke-cluster-1-default-pool-6d42fa0a-1ckn because it does not exist in the cloud provider   node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn event: DeletingNode
3m45s       Normal    RemovingNode                                                                                             node/gke-cluster-1-default-pool-6d42fa0a-1ckn   Node gke-cluster-1-default-pool-6d42fa0a-1ckn event: Removing Node gke-cluster-1-default-pool-6d42fa0a-1ckn from Controller
</code></pre>
<h3>Conclusion</h3>
<p>By default, <code>kube-system</code> pods prevent CA from removing nodes on which they are running. Users can manually add <code>PDBs </code> for the <code>kube-system</code> pods that can be safely rescheduled elsewhere. It can be achieved using:</p>
<pre><code>kubectl create poddisruptionbudget &lt;pdb name&gt; --namespace=kube-system --selector app=&lt;app name&gt; --max-unavailable 1
</code></pre>
<p>List of possible reasons why <code>CA</code> won't autoscale can be found in <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="noreferrer">Cluster Autoscaler - Frequently Asked Questions</a>.</p>
<p>To verify which pods could still block <code>CA</code> downscale, you can use <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#viewing_events" rel="noreferrer">Autoscaler Events</a>.</p>
