<p>I have two nodepools in my AKS cluster; the default nodepool and an 'application' nodepool. I use the default nodepool for services like Airflow, and the application nodepool to run ETL jobs. However, the application nodepool never scales to zero, even when I do not schedule any ETL jobs for many hours.</p>
<p>I fail to understand why. Is there anyone who has any suggestions for the rootcause of the issue?</p>
<p>The cluster is depolyed using Terraform. The autoscaler is configured as foloows:</p>
<pre><code>auto_scaler_profile {
    # (Optional) Maximum number of seconds the cluster autoscaler waits for pod termination when trying to scale down a node. Defaults to 600.
    max_graceful_termination_sec = 180
    # (Optional) How long after the scale up of AKS nodes the scale down evaluation resumes. Defaults to 10m.)
    scale_down_delay_after_add = &quot;3m&quot;
    # - (Optional) How long a node should be unneeded before it is eligible for scale down. Defaults to 10m.
    scale_down_unneeded = &quot;3m&quot;
    # (Optional) If true cluster autoscaler will never delete nodes with pods from kube-system (except for DaemonSet or mirror pods). Defaults to true.
    skip_nodes_with_system_pods = false
  }
</code></pre>
<p>and the application nodepool is defined as:</p>
<pre><code>resource &quot;azurerm_kubernetes_cluster_node_pool&quot; &quot;main&quot; {
  name                  = &quot;application&quot;
  kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id
  vm_size               = &quot;Standard_B4ms&quot;
  enable_auto_scaling   = true
  min_count             = 0
  max_count             = 2
  max_pods              = 15

  node_labels = {
    &quot;type&quot; = &quot;application&quot;
  }

}
</code></pre>
<hr />
<p>Below are some relevant details about the AKS cluster:</p>
<pre class="lang-bash prettyprint-override"><code>k top nodes

NAME                                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
aks-application-XXXXXXXX-vmss000000   55m          1%     1579Mi          12%       
aks-default-XXXXXXXX-vmss000000       677m         17%    7783Mi          61%  
</code></pre>
<pre class="lang-bash prettyprint-override"><code>az aks nodepool show \
--resource-group &lt;my-rg&gt; \
--cluster-name &lt;my-cluster&gt; \
--name application \
--query &quot;{min: minCount, max: maxCount}&quot;

{
  &quot;max&quot;: 2,
  &quot;min&quot;: 0
}
</code></pre>
<pre class="lang-bash prettyprint-override"><code>az aks show \
--resource-group &lt;my-rg&gt; \
--name &lt;my-cluster&gt; \
--query autoScalerProfile

{
  &quot;balanceSimilarNodeGroups&quot;: &quot;false&quot;,
  &quot;expander&quot;: &quot;random&quot;,
  &quot;maxEmptyBulkDelete&quot;: &quot;10&quot;,
  &quot;maxGracefulTerminationSec&quot;: &quot;180&quot;,
  &quot;maxNodeProvisionTime&quot;: &quot;15m&quot;,
  &quot;maxTotalUnreadyPercentage&quot;: &quot;45&quot;,
  &quot;newPodScaleUpDelay&quot;: &quot;0s&quot;,
  &quot;okTotalUnreadyCount&quot;: &quot;3&quot;,
  &quot;scaleDownDelayAfterAdd&quot;: &quot;3m&quot;,
  &quot;scaleDownDelayAfterDelete&quot;: &quot;10s&quot;,
  &quot;scaleDownDelayAfterFailure&quot;: &quot;3m&quot;,
  &quot;scaleDownUnneededTime&quot;: &quot;3m&quot;,
  &quot;scaleDownUnreadyTime&quot;: &quot;20m&quot;,
  &quot;scaleDownUtilizationThreshold&quot;: &quot;0.5&quot;,
  &quot;scanInterval&quot;: &quot;10s&quot;,
  &quot;skipNodesWithLocalStorage&quot;: &quot;true&quot;,
  &quot;skipNodesWithSystemPods&quot;: &quot;false&quot;
}
</code></pre>
<pre class="lang-bash prettyprint-override"><code>k get pods  --sort-by=&quot;{.spec.nodeName}&quot; -A -o wide                                                                                                          
NAMESPACE      NAME                                  READY   STATUS    RESTARTS   AGE     NODE                                 
kube-system    azure-ip-masq-agent-XXXXX             1/1     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    metrics-server-XXXXXXXXXX-XXXXX       2/2     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    metrics-server-XXXXXXXXXX-XXXXX       2/2     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    kube-proxy-XXXXX                      1/1     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    csi-blob-node-XXXXX                   3/3     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    csi-azurefile-node-XXXXX              3/3     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    csi-azuredisk-node-XXXXX              3/3     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    cloud-node-manager-XXXXX              1/1     Running   0          3d17h   aks-application-XXXXXXXX-vmss000000
kube-system    cloud-node-manager-XXXXX              1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-pgbouncer-XXXXXXXXXX-XXXXX    2/2     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-triggerer-XXXXXXXXX-XXXXX     1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-webserver-XXXXXXXXX-XXXXX     1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-scheduler-XXXXXXXXX-XXXXX     2/2     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
kube-system    azure-ip-masq-agent-XXXXX             1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-postgresql-0                  1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
kube-system    coredns-XXXXXXXXXX-XXXXX              1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
kube-system    coredns-XXXXXXXXXX-XXXXX              1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
kube-system    coredns-autoscaler-XXXXXXXXXX-XXXXX   1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
airflow-prod   airflow-statsd-XXXXXXXX-XXXXX         1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
kube-system    csi-azuredisk-node-XXXXX              3/3     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
kube-system    csi-azurefile-node-XXXXX              3/3     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
kube-system    csi-blob-node-XXXXX                   3/3     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
kube-system    konnectivity-agent-XXXXXXXXXX-XXXXX   1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
kube-system    konnectivity-agent-XXXXXXXXXX-XXXXX   1/1     Running   0          3d17h   aks-default-XXXXXXXX-vmss000000
kube-system    kube-proxy-XXXXX                      1/1     Running   0          3d21h   aks-default-XXXXXXXX-vmss000000
</code></pre>
<p><a href="https://i.stack.imgur.com/GeTQT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/GeTQT.png" alt="enter image description here" /></a></p>

Florian
<p>Some system pods block node removal because there is no node affinity on your system pods.</p>
<p>You can remedy this by deploying a dedicated system node pool.</p>
