<p>our autoscaling (horizontal and vertical) works pretty fine, except the downscaling is not working somehow (yeah, we checked the usual suspects like <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#i-have-a-couple-of-nodes-with-low-utilization-but-they-are-not-scaled-down-why" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#i-have-a-couple-of-nodes-with-low-utilization-but-they-are-not-scaled-down-why</a> ).</p>
<p>Since we want to save resources and have pods which are not ultra-sensitive, we are setting following</p>
<p><strong>Deployment</strong></p>
<pre class="lang-yaml prettyprint-override"><code>replicas: 1
</code></pre>
<p><strong>PodDisruptionBudget</strong></p>
<pre class="lang-yaml prettyprint-override"><code>minAvailable: 1
</code></pre>
<p><strong>HorizontalPodAutoscaler</strong></p>
<pre class="lang-yaml prettyprint-override"><code>minReplicas: 1
maxReplicas: 10
</code></pre>
<p>But it seems now that this is the problem that the autoscaler is not scaling down the nodes (even though the node is only used by 30% by CPU + memory and we have other nodes which have absolutely enough memory + cpu to move these pods).</p>
<p>Is it possible in general that the auto scaler starts an extra pod on the free node and removes the old pod from the old node?</p>

BvuRVKyUVlViVIc7
<blockquote>
<p>Is it possible in general that the auto scaler starts an extra pod on the free node and removes the old pod from the old node?</p>
</blockquote>
<p>Yes, that should be possible in general, but in order for the cluster autoscaler to remove a node, it must be possible to move <strong>all pods</strong> running on the node somewhere else.</p>
<p>According to <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node" rel="nofollow noreferrer">docs</a> there are a few type of pods that are not movable:</p>
<blockquote>
<ul>
<li>Pods with restrictive PodDisruptionBudget.</li>
<li>Kube-system pods that:
<ul>
<li>are not run on the node by default</li>
<li>don't have a pod disruption budget set or their PDB is too restrictive &gt;(since CA 0.6).</li>
</ul>
</li>
<li>Pods that are not backed by a controller object (so not created by &gt;deployment, replica set, job, stateful set etc).</li>
<li>Pods with local storage.</li>
<li>Pods that cannot be moved elsewhere due to various constraints (lack of &gt;resources, non-matching node selectors or affinity, matching anti-affinity, etc)</li>
<li>Pods that have the following annotation set:
<code>cluster-autoscaler.kubernetes.io/safe-to-evict: &quot;false</code></li>
</ul>
</blockquote>
<p>You could check the cluster autoscaler logs, they may provide a hint to why no scale in happens:</p>
<pre><code>kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler
</code></pre>
<p>Without having more information about your setup it is hard to guess what is going wrong, but unless you are using local storage, node selectors or affinity/anti-affinity rules etc Pod disruption policies is a likely candidate. Even if you are not using them explicitly they can still prevent node scale in if they there are pods in the <code>kube-system</code> namespace that are missing pod disruption policies (See <a href="https://stackoverflow.com/a/65811347/7146596">this answer</a> for an example of such a scenario in GKE)</p>
