title: Cluster Autoscaler on EKS during scale-down ignores pod annotation & node annotations
body: <p>I'm using Cluster-autoscaler (v1.18.3) on Amazon EKS (Cluster version 1.18). Every time we've an increase in load the cluster-autoscaler adds new nodes to deal with the extra load. But when the load decreases cluster-autoscaler need to reduce the number of nodes back to minimum.
  Since I've set on all the Statefulsets pod template the annotation <code>&quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;</code> I do not expect the cluster-autoscaler to remove any node that contains any pods having this annotation. I can also see cluster-autoscaler logs showing <code>1 cluster.go:168] Fast evaluation: node ip-172-30-59-87.eu-west-2.compute.internal cannot be removed: pod annotated as not safe to evict present: jenkins-0</code> but still cluster-autoscaler removes this exact same nodes (that contain the statefulset pods).</p>
  <p>I've noticed cluster-autoscaler is always removing oldest nodes from the cluster when scaling in.</p>
  <p>I also tried setting node annotations of &quot;cluster-autoscaler.kubernetes.io/scale-down-disabled&quot;: &quot;true&quot; on the nodes which contain those important Jenkins statefulset pods. But still cluster-autoscaler will always remove the oldest nodes in the cluster. No matter what node / pod annotations I set it will always remove the oldest nodes.</p>
  <p><strong>The issue</strong></p>
  <p>I don't want nodes that contain Statefulset pods which I've annotated as <code>&quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;</code> to be removed. The new nodes that were launched to deal with the load should be the ones that should be removed.</p>
  <p>I also do not expect cluster-autoscaler to remove a node that has annotation <code>&quot;cluster-autoscaler.kubernetes.io/scale-down-disabled&quot;: &quot;true&quot;</code> to be removed.</p>
  <p>The oldest nodes are the ones that are always removed. Why is this happening? I've opened an issue/bug 3 weeks before opening this post <a href="https://github.com/kubernetes/autoscaler/issues/3978" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/issues/3978</a> but this hasn't been commented or picked up by anyone.</p>
  <p><strong>How I've setup cluster-autoscaler?</strong></p>
  <p>Deployed cluster autoscaler using AWS Documentation <a href="https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html</a> or <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md</a>.</p>
  <p><strong>How to produce the issue?</strong></p>
  <p>Ensure you've setup an EKS cluster with 2 node groups (One per Az) with 1 node min in each node group.</p>
  <ol>
  <li>Install Jenkins statefulset using helm chart <a href="https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins" rel="nofollow noreferrer">https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins</a>.</li>
  <li>Annotate pods in values file</li>
  </ol>
  <pre><code>...
  controller:
    podAnnotations:
      &quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;: &quot;false&quot;
  ...
  </code></pre>
  <p>Note: I've just mentioned the podAnnotations here as the values file is more than 200 lines of code setting different things like pvc, config as code, plugins etc.</p>
  <ol start="3">
  <li>Deploy cluster-autoscaler from AWS documentation.</li>
  <li>Now to produce some load on the cluster create a simple Deployment of Nginx (3GBRAM, 2CPU) &amp; Scale out to 50 replicas. Cluster Autoscaler will add new EC2 nodes.</li>
  <li>Now scale down replicas to 0 and wait for 10 mins for Cluster Autoscaler to scale down.
  Cluster Autoscaler will log output like <code>1 cluster.go:168] Fast evaluation: node ip-172-30-59-87.eu-west-2.compute.internal cannot be removed: pod annotated as not safe to evict present: jenkins-0</code> but still remove this exact node.</li>
  </ol>
  <p><strong>Tried Configurations</strong></p>
  <p>Cluster-autoscaler images tried: 1.18.3, 1.18.2 on Kubernetes 1.18 cluster. Also tried images 1.19.2 on Kubernetes 1.19 cluster with exact same results.</p>
  <p>Ran cluster-autoscaler with configuration:</p>
  <pre><code>spec: 
    containers: 
    - command: 
      - ./cluster-autoscaler 
      - --v=4 
      - --stderrthreshold=info 
      - --cloud-provider=aws 
      - --skip-nodes-with-local-storage=false 
      - --expander=least-waste 
      - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/xxxmyclusternamexxx 
      - --balance-similar-node-groups 
      - --skip-nodes-with-system-pods=false
  </code></pre>
  <p>also tried:</p>
  <pre><code>spec: 
    containers: 
     - command: 
       - ./cluster-autoscaler 
       - --v=4 
       - --stderrthreshold=info 
       - --cloud-provider=aws 
       - --expander=least-waste 
       - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/xxxxmyclusternamexxxxx 
       - --balance-similar-node-groups
  </code></pre>