<p>we are running <strong>6</strong> nodes in K8s cluster. Out of 6, 2 of them running RabbitMQ, Redis &amp; Prometheus we have used node-selector &amp; cordon node so no other pods schedule on that particular nodes.</p>
<p>On renaming other 4 nodes application PODs run, we have around 18-19 micro services.
For GKE there is one open issue in K8s official repo regarding auto scale down: <a href="https://github.com/kubernetes/kubernetes/issues/69696#issuecomment-651741837" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/69696#issuecomment-651741837</a> automatically however people are suggesting approach of setting <strong>PDB</strong> and we that tested on Dev/Stag.</p>
<p>What we are looking for now is to fix PODs on particular <strong>node pool</strong> which do not scale, as we are running single replicas of some services.</p>
<p>As of now, we thought of using and apply affinity to those services which are running with <strong>single replicas</strong> and no requirement of <strong>scaling</strong>.</p>
<p>while for scalable services we won't specify any type of rule so by default K8s scheduler will schedule pod across different nodes, so this way if any node scale down we dont face any downtime for <strong>single</strong> running replica service.</p>
<p>Affinity example :</p>
<pre><code>affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: do-not-scale
                operator: In
                values:
                - 'true'
</code></pre>
<p>We are planning to use affinity type <code>preferredDuringSchedulingIgnoredDuringExecution</code> instead of <code>requiredDuringSchedulingIgnoredDuringExecution</code>.</p>
<p>Note : Here K8s is not creating new replica first on another node during node drain (scaledown of any node) as we are running single replicas with rolling update &amp; minAvailable: 25% strategy.</p>
<p>Why: If PodDisruptionBudget is not specified and we have a deployment with one replica, the pod will be terminated and then a new pod will be scheduled on a new node.</p>
<p>To make sure the application will be available during the node draining process we have to specify PodDisruptionBudget and create more replicas. If we have 1 pod with minAvailable: 30% it will refuse to drain node (scaledown).</p>
<p>Please point out a mistake if you are seeing anything wrong &amp; suggest better option.</p>

chagan
<p>First of all, defining <code>PodDisruptionBudget</code> makes not much sense whan having only one replica. <code>minAvailable</code> expressed as a percentage is rounded up to an integer as it represents the minimum number of <code>Pods</code> which need to be available all the time.</p>
<p>Keep in mind that you have no guarantee for any High Availability when launching only one-replica <code>Deployments</code>.</p>
<blockquote>
<p>Why: If PodDisruptionBudget is not specified and we have a deployment
with one replica, the pod will be terminated and then a new pod will
be scheduled on a new node.</p>
</blockquote>
<p>If you didn't explicitely define in your <code>Deployment</code>'s <code>spec</code> the value of <code>maxUnavailable</code>, by default it is set to 25%, which being rounded up to an integer (representing number of <code>Pods</code>/<code>replicas</code>) equals <code>1</code>. It means that 1 out of 1 replicas is allowed to be unavailable.</p>
<blockquote>
<p>If we have 1 pod with minAvailable: 30% it will refuse to drain node
(scaledown).</p>
</blockquote>
<p>Single replica with <code>minAvailable: 30%</code> is rounded up to <code>1</code> anyway. <code>1/1</code> should be still up and running so <code>Pod</code> cannot be evicted and node cannot be drained in this case.</p>
<p>You can try the following solution however I'm not 100% sure if it will work when your <code>Pod</code> is re-scheduled to another node due to it's eviction from the one it is currently running on.</p>
<p>But if you re-create your <code>Pod</code> e.g. because you update it's image to a new version, you can guarantee that at least one replica will be still up and running (old <code>Pod</code> won't be deleted unless the new one enters <code>Ready</code> state) by setting <code>maxUnavailable: 0</code>. As per the <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment" rel="nofollow noreferrer">docs</a>, by default it is set to <code>25%</code> which is rounded up to <code>1</code>. So by default you allow that one of your replicas (which in your case happens to be <code>1/1</code>) becomes unavailable during the rolling update. If you set it to zero, it won't allow the old <code>Pod</code> to be deleted unless the new one becomes <code>Ready</code>. At the same time <code>maxSurge: 2</code> allows that 2 replicas temporarily exist at the same time during the update.</p>
<p>Your <code>Deployment</code> definition may begin as follows:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0 👈
      maxSurge: 2
  selector:
  ...
</code></pre>
<p>Compare it with <a href="https://stackoverflow.com/a/46377159/11714114">this answer</a>, provided by <a href="https://stackoverflow.com/users/225016/mdaniel">mdaniel</a>, where I originally found it.</p>
