<p>Im looking for the solution that will scale out pods automatically when the nodes join the cluster and scale in back when the nodes are deleted. 
We are running WebApp on the nodes and this require graceful pod eviction/termination when the node is scheduled to be disconnected.
I was checking the option of using the DaemonSet but since we are using Kops for the cluster rolling update it ignores DaemonSets evictions (flag "--ignore-daemionset" is not supported).
As a result the WebApp "dies" with the node which is not acceptable for our application.
The ability of HorizontalPodAutoscaler to overwrite the amount of replicas which are set in the deployment yaml could solve the problem.
I want to find the way to change the min/maxReplicas in HorizontalPodAutoscaler yaml dynamically based on the amount of nodes in the cluster.</p>

<pre><code>spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: MyWebApp
  minReplicas: "Num of nodes in the cluster"
  maxReplicas: "Num of nodes in the cluster"
</code></pre>

<p>Any ideas how to get the number of nodes and update HorizontalPodAutoscaler yaml in the cluster accordingly? Or any other solutions for the problem? </p>

<p>Have you tried usage of <code>nodeSelector</code> spec in daemonset yaml.
So <strong>if you have nodeselector set in yaml and just before drain if you remove the nodeselector label value from the node the daemonset should scale down gracefully</strong> also same when you add new node to cluster label the node with custom value and deamonset will scale up.</p>

<p>This works for me so you can try this and confirm with Kops </p>

<p>First : Label all you nodes with a custom label you will always have on your cluster </p>

<p>Example: </p>

<pre><code>kubectl label nodes k8s-master-1 mylabel=allow_demon_set  
kubectl label nodes k8s-node-1 mylabel=allow_demon_set
kubectl label nodes k8s-node-2 mylabel=allow_demon_set
kubectl label nodes k8s-node-3 mylabel=allow_demon_set
</code></pre>

<p>Then to your daemon set yaml add node selector.</p>

<p>Example.yaml used as below : <strong>Note</strong>  <strong>added nodeselctor field</strong> </p>

<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      nodeSelector:
        mylabel: allow_demon_set
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
</code></pre>

<p>So nodes are labeled as below</p>

<pre><code>$ kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE   VERSION   LABELS
k8s-master-1   Ready    master   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master-1,kubernetes.io/os=linux,mylable=allow_demon_set,node-role.kubernetes.io/master=
k8s-node-1     Ready    &lt;none&gt;   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-1,kubernetes.io/os=linux,mylable=allow_demon_set
k8s-node-2     Ready    &lt;none&gt;   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-2,kubernetes.io/os=linux,mylable=allow_demon_set
k8s-node-3     Ready    &lt;none&gt;   9d    v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-3,kubernetes.io/os=linux,mylable=allow_demon_set
</code></pre>

<p>Once you have correct yaml start the daemon set using it </p>

<pre><code>$ kubectl create -f Example.yaml

$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-jrgl6   1/1     Running   0          20s   10.244.3.19   k8s-node-3     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-rgcm2   1/1     Running   0          20s   10.244.0.6    k8s-master-1   &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wccr9   1/1     Running   0          20s   10.244.1.14   k8s-node-1     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wxq5v   1/1     Running   0          20s   10.244.2.33   k8s-node-2     &lt;none&gt;           &lt;none&gt;

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    &lt;none&gt;        443/TCP   9d    &lt;none&gt;

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE   CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   4         4         4       4            4           mylable=allow_demon_set   20s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch
</code></pre>

<p>Then <strong>before draining a node we can just remove the custom label from node</strong> and the pod-should scale down gracefully and then drain the node.</p>

<p><strong><code>$ kubectl label nodes k8s-node-3 mylabel-</code></strong></p>

<p><strong>Check the daemonset and it should scale down</strong></p>

<pre><code>ubuntu@k8s-kube-client:~$ kubectl get all -o wide
NAME                              READY   STATUS        RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-jrgl6   0/1     Terminating   0          2m36s   10.244.3.19   k8s-node-3     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-rgcm2   1/1     Running       0          2m36s   10.244.0.6    k8s-master-1   &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wccr9   1/1     Running       0          2m36s   10.244.1.14   k8s-node-1     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wxq5v   1/1     Running       0          2m36s   10.244.2.33   k8s-node-2     &lt;none&gt;           &lt;none&gt;

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    &lt;none&gt;        443/TCP   9d    &lt;none&gt;

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE     CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   3         3         3       3            3           mylable=allow_demon_set   2m36s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch
</code></pre>

<p>Now again add the <strong>label to new node with same custom label when it is added to cluster and the deamonset will scale up</strong> </p>

<p><strong><code>$ kubectl label nodes k8s-node-3 mylable=allow_demon_set</code></strong></p>

<pre><code>ubuntu@k8s-kube-client:~$ kubectl get all -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
pod/fluentd-elasticsearch-22rsj   1/1     Running   0          2s      10.244.3.20   k8s-node-3     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-rgcm2   1/1     Running   0          5m28s   10.244.0.6    k8s-master-1   &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wccr9   1/1     Running   0          5m28s   10.244.1.14   k8s-node-1     &lt;none&gt;           &lt;none&gt;
pod/fluentd-elasticsearch-wxq5v   1/1     Running   0          5m28s   10.244.2.33   k8s-node-2     &lt;none&gt;           &lt;none&gt;

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    &lt;none&gt;        443/TCP   9d    &lt;none&gt;

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR             AGE     CONTAINERS              IMAGES                                         SELECTOR
daemonset.apps/fluentd-elasticsearch   4         4         4       4            4           mylable=allow_demon_set   5m28s   fluentd-elasticsearch   quay.io/fluentd_elasticsearch/fluentd:v2.5.2   name=fluentd-elasticsearch
</code></pre>

<p><strong>Kindly confirm if this what you want to do and works with kops</strong></p>
