<p>Say I have 100 running pods with an HPA set to <code>min=100</code>, <code>max=150</code>. Then I change the HPA to <code>min=50</code>, <code>max=105</code> (e.g. max is still above current pod count). Should k8s immediately initialize new pods when I change the HPA? I wouldn't think it does, but I seem to have observed this today.</p>

L P
<p>First, as mentioned in the comments, in your specific case some pods will be terminated if usage metrics are below utilization target, no new pods will be created.</p>
<p>Second thing it's absolutely normal that is takes some time to scale down replicas - <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#default-behavior" rel="nofollow noreferrer">it's because the
<code>stabilizationWindowSeconds</code> parameter is by default set to <code>300</code></a>:</p>
<blockquote>
<pre class="lang-yaml prettyprint-override"><code>behavior:
scaleDown:
   stabilizationWindowSeconds: 300
   policies:
   - type: Percent
     value: 100
     periodSeconds: 15
</code></pre>
</blockquote>
<p>So, if you have running HPA with configuration (min=100, max=150) for a long time, and you have changed to min=50, max=105, then after 300 seconds (5 minutes) your replicas will be scaled down to the 50 replicas.</p>
<p>Good explanation about how exactly <code>stabilizationWindowSeconds</code> works is in <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md#story-5-stabilization-before-scaling-down" rel="nofollow noreferrer">this document</a>:</p>
<blockquote>
<h4>Story 5: Stabilization before scaling down</h4>
<p>This mode is used when the user expects a lot of flapping or does not want to scale down pods too early expecting some late load spikes.</p>
<p>Create an HPA with the following behavior:</p>
<pre class="lang-yaml prettyprint-override"><code>behavior:
 scaleDown:
   stabilizationWindowSeconds: 600
   policies:
   - type: Pods
     value: 5
</code></pre>
<p>i.e., the algorithm will:</p>
<ul>
<li>gather recommendations for 600 seconds  <em>(default: 300 seconds)</em></li>
<li>pick the largest one</li>
<li>scale down no more than 5 pods per minute</li>
</ul>
<p>Example for  <code>CurReplicas = 10</code>  and HPA controller cycle once per a minute:</p>
<ul>
<li>First 9 minutes the algorithm will do nothing except gathering recommendations. Let's imagine that we have the following recommendations</li>
</ul>
<p>recommendations = [10, 9, 8, 9, 9, 8, 9, 8, 9]</p>
<ul>
<li>On the 10th minute, we'll add one more recommendation (let it me  <code>8</code>):</li>
</ul>
<p>recommendations = [10, 9, 8, 9, 9, 8, 9, 8, 9, 8]</p>
<p>Now the algorithm picks the largest one  <code>10</code>. Hence it will not change number of replicas</p>
<ul>
<li>On the 11th minute, we'll add one more recommendation (let it be  <code>7</code>) and removes the first one to keep the same amount of recommendations:</li>
</ul>
<p>recommendations = [9, 8, 9, 9, 8, 9, 8, 9, 8, 7]</p>
<p>The algorithm picks the largest value  <code>9</code>  and changes the number of replicas  <code>10 -&gt; 9</code></p>
</blockquote>
<p>Another thing is that it depends which Kubernetes version, which <code>apiVersion</code> for the autoscaling are you using and which Kuberntes solution are you using. The behaviour could vary - check <a href="https://github.com/kubernetes/kubernetes/issues/78761" rel="nofollow noreferrer">this topic on GitHub</a> with a bug reports.</p>
<p>If you want to have scale down done immediately (not recommended in the production), you can setup following:</p>
<pre class="lang-yaml prettyprint-override"><code>behavior:
    scaleDown:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 1
</code></pre>
<p>Also check:</p>
<ul>
<li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">Horizontal Pod Autoscaling</a> in particular <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#example-change-downscale-stabilization-window" rel="nofollow noreferrer">Example: change downscale stabilization window</a> and <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#example-limit-scale-down-rate" rel="nofollow noreferrer">Example: limit scale down rate</a></li>
<li><a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md#configurable-scale-updown-velocity-for-hpa" rel="nofollow noreferrer">Configurable scale up/down velocity for HPA</a></li>
</ul>
