<p>In a nutshell, most of our apps are configured with the following <code>strategy</code> in the Deployment - </p>

<pre><code>  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
</code></pre>

<p>The Horizonatal Pod Autoscaler is configured as so </p>

<pre><code>spec:
  maxReplicas: 10
  minReplicas: 2
</code></pre>

<p>Now when our application was redeployed, instead of running a rolling update, it instantly terminated 8 of our pods and dropped the number of pods to <code>2</code> which is the min number of replicas available. This happened in a fraction of a second as you can see here.</p>

<p><a href="https://i.stack.imgur.com/V7AVN.png" rel="noreferrer"><img src="https://i.stack.imgur.com/V7AVN.png" alt="enter image description here"></a></p>

<p>Here is the output of <code>kubectl get hpa</code> - </p>

<p><a href="https://i.stack.imgur.com/ehlyV.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ehlyV.png" alt="enter image description here"></a></p>

<p>As <code>maxUnavailable</code> is 25%, shouldn't only about 2-3 pods go down at max ? Why did so many pods crash at once ? It seems as though rolling update is useless if it works this way.</p>

<p>What am I missing ?</p>

<p>In our case we added the <code>replicas</code> field a while ago and forgot to remove it when we added the HPA. The HPA does not play nice with the <code>replicas</code> field during deployments, so if you have a HPA remove the <code>replicas</code> field. See <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling</a></p>
<blockquote>
<p>When an HPA is enabled, it is recommended that the value of spec.replicas of the Deployment and / or StatefulSet be removed from their manifest(s). If this isn't done, any time a change to that object is applied, for example via kubectl apply -f deployment.yaml, this will instruct Kubernetes to scale the current number of Pods to the value of the spec.replicas key. This may not be desired and could be troublesome when an HPA is active.</p>
</blockquote>
<blockquote>
<p>Keep in mind that the removal of spec.replicas may incur a one-time degradation of Pod counts as the default value of this key is 1 (reference Deployment Replicas). Upon the update, all Pods except 1 will begin their termination procedures. Any deployment application afterwards will behave as normal and respect a rolling update configuration as desired.</p>
</blockquote>
