<p>In a nutshell, most of our apps are configured with the following <code>strategy</code> in the Deployment - </p>

<pre><code>  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
</code></pre>

<p>The Horizonatal Pod Autoscaler is configured as so </p>

<pre><code>spec:
  maxReplicas: 10
  minReplicas: 2
</code></pre>

<p>Now when our application was redeployed, instead of running a rolling update, it instantly terminated 8 of our pods and dropped the number of pods to <code>2</code> which is the min number of replicas available. This happened in a fraction of a second as you can see here.</p>

<p><a href="https://i.stack.imgur.com/V7AVN.png" rel="noreferrer"><img src="https://i.stack.imgur.com/V7AVN.png" alt="enter image description here"></a></p>

<p>Here is the output of <code>kubectl get hpa</code> - </p>

<p><a href="https://i.stack.imgur.com/ehlyV.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ehlyV.png" alt="enter image description here"></a></p>

<p>As <code>maxUnavailable</code> is 25%, shouldn't only about 2-3 pods go down at max ? Why did so many pods crash at once ? It seems as though rolling update is useless if it works this way.</p>

<p>What am I missing ?</p>

<p>After looking at this question, I decided to try this with test Environment where I wanted to check If it doesn't work.</p>

<p>I have setup the <code>metrics-server</code> to fetch the metrics server and set a HPA. I have followed the following steps to setup the HPA and deployment:</p>

<p><a href="https://stackoverflow.com/questions/53725248/how-to-enable-kubeapi-server-for-hpa-autoscaling-metrics/53727101#53727101">How to Enable KubeAPI server for HPA Autoscaling Metrics</a></p>

<p>Once, I have working HPA and max <code>10 pods</code> running on system, I have updated the images using:</p>

<pre><code>[root@ip-10-0-1-176 ~]# kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   49%/50%   1         10        10         87m

[root@ip-10-0-1-176 ~]# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
load-generator-557649ddcd-6jlnl   1/1     Running   0          61m
php-apache-75bf8f859d-22xvv       1/1     Running   0          91s
php-apache-75bf8f859d-dv5xg       1/1     Running   0          106s
php-apache-75bf8f859d-g4zgb       1/1     Running   0          106s
php-apache-75bf8f859d-hv2xk       1/1     Running   0          2m16s
php-apache-75bf8f859d-jkctt       1/1     Running   0          2m46s
php-apache-75bf8f859d-nlrzs       1/1     Running   0          2m46s
php-apache-75bf8f859d-ptg5k       1/1     Running   0          106s
php-apache-75bf8f859d-sbctw       1/1     Running   0          91s
php-apache-75bf8f859d-tkjhb       1/1     Running   0          55m
php-apache-75bf8f859d-wv5nc       1/1     Running   0          106s
[root@ip-10-0-1-176 ~]# kubectl set image deployment php-apache php-apache=hpa-example:v1 --record
deployment.extensions/php-apache image updated

[root@ip-10-0-1-176 ~]# kubectl get pods
NAME                              READY   STATUS              RESTARTS   AGE
load-generator-557649ddcd-6jlnl   1/1     Running             0          62m
php-apache-75bf8f859d-dv5xg       1/1     Terminating         0          2m40s
php-apache-75bf8f859d-g4zgb       1/1     Terminating         0          2m40s
php-apache-75bf8f859d-hv2xk       1/1     Terminating         0          3m10s
php-apache-75bf8f859d-jkctt       1/1     Running             0          3m40s
php-apache-75bf8f859d-nlrzs       1/1     Running             0          3m40s
php-apache-75bf8f859d-ptg5k       1/1     Terminating         0          2m40s
php-apache-75bf8f859d-sbctw       0/1     Terminating         0          2m25s
php-apache-75bf8f859d-tkjhb       1/1     Running             0          56m
php-apache-75bf8f859d-wv5nc       1/1     Terminating         0          2m40s
php-apache-847c8ff9f4-7cbds       1/1     Running             0          6s
php-apache-847c8ff9f4-7vh69       1/1     Running             0          6s
php-apache-847c8ff9f4-9hdz4       1/1     Running             0          6s
php-apache-847c8ff9f4-dlltb       0/1     ContainerCreating   0          3s
php-apache-847c8ff9f4-nwcn6       1/1     Running             0          6s
php-apache-847c8ff9f4-p8c54       1/1     Running             0          6s
php-apache-847c8ff9f4-pg8h8       0/1     Pending             0          3s
php-apache-847c8ff9f4-pqzjw       0/1     Pending             0          2s
php-apache-847c8ff9f4-q8j4d       0/1     ContainerCreating   0          4s
php-apache-847c8ff9f4-xpbzl       0/1     Pending             0          1s
</code></pre>

<p>Also, I have kept job in background which pushed the <code>kubectl get pods</code> output every second in a file. At no time till all images are upgraded, number of pods never went below 8. </p>

<p>I believe you need to check how you're setting up your rolling upgrade. Are you using deployment or replicaset? I have kept the <code>rolling update</code> strategy same as you <code>maxUnavailable: 25%</code> and <code>maxSurge: 25%</code> with deployment and it is working well for me.</p>
