<p>I have a Kubernetes cluster set up in DigitalOcean. The cluster is configured to auto-scale using HPA(Horizontal Pod Autoscaler). I want to prevent termination of a pod that got scaled up in the last 1 hour to avoid thrashing and saving the bill.
Following are the two reasons for the same:</p>

<ol>
<li>Due to unpredictable traffic, sometimes new pods scale up and down multiple times in an hour. Because of the nature of the application, 50-60 new users need a new pod to handle the traffic. </li>
<li>DigitalOcean droplets are charged per hour. Even if the droplet was up for 15 minutes, They would charge it for an hour. So, sometimes we are paying for 5 droplets in an hour which could have been paid for just 1 droplet. </li>
</ol>

<p>From the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="noreferrer">documentation</a>, I could not find anything related to this. Any hack for the same would be helpful. </p>

Nitesh
<p>Yes we can do this. I currently doing this experimentation almost related to your question.</p>
<p>Try to find Following things while autoscaling.</p>
<ol>
<li>Time taken for HPA to calculate Replica needed</li>
<li>Time taken for pod to Spin up.</li>
<li>Time taken to Droplet spin up.</li>
<li>Time taken for pods spin down.</li>
<li>Time taken to Droplet Spin down.</li>
</ol>
<p><strong>Case 1: Time taken for HPA to calculate Replica needed  (HPA)</strong></p>
<p>HPA detect the changes, As soon as get metrics immediately or atleast within 15 secs. Depends on <code>horizontal-pod-autoscaler-sync-period</code> By default it is set to 15 secs. As soon HPA get Metric, it calculates Replica Needed.</p>
<p><strong>Case 2: Time taken for pod to Spin up. (HPA)</strong></p>
<p>As soon as HPA calculate Desired Replicas, Pods start spin up. But it depends on <code>ScaleUp Policy</code>. You can set this as per your use case.And also depend on Droplet available, cluster autoscaler</p>
<p>For Example:
You can tell HPA, Hey, please spin up 4 pods in 15 secs OR Spin up 100 % of current available pods in 20 secs.</p>
<p>Now HPA, will take decision to select anyone policy, which make more impact(Most changes in replica count). If <code>100% pods &gt; 4 pods</code> ,Second policy takeover, otherwise first Policy can take over.
Process repeats until reach the desried replica.</p>
<p>If you need scaled up Pod count immediately, you set policy as spin up 100 % pods in 1 secs, hence it try to spin up 100 % of current replica count in every secs until match the Desired  Replica count.</p>
<p><strong>Case 3: Time taken to Droplet spin up. (Cluster Autoscaler)</strong></p>
<p>Time Taken For:</p>
<ul>
<li>Cluster autoscaler to detect pending pods and start spinning droplet: <code>1 min 05 secs</code> (approx)</li>
<li>Droplet spin up , but Not Ready State: <code>1 min 20 secs</code></li>
<li>Droplet to each READY STATE: <code>10 - 20 secs</code></li>
</ul>
<p><code>Total Time taken to droplet Available: 2 min 40 secs (approx)</code></p>
<p><strong>Case 4: Time taken for pod to spin down. (HPA)</strong></p>
<p>It depends on ScalDown Policy, as like as Case 2.</p>
<p><strong>Case 5: Time taken to Droplet Spin down. (Cluster Autoscaler)</strong></p>
<p>After all the Target pods terminated from the Droplet(Time taken depends on case 4).</p>
<p>Digital Ocean set Taints to node like <code>DeletionCandidate...=&lt;timestamp&gt;:NopreferSchedule</code></p>
<p>Ten mins from taint set, droplet starts spin down.</p>
<p><strong>Conclusion:</strong></p>
<p>If you need Node for one hour to stay alive (utilize as max because of hourly charge)
And Not cross one hour(if above 1 hr, it billed as 2 hr)</p>
<p>You can set, StabilizatioWindowSeconds = 1 hr - DigitalOcean Time Interval to delete</p>
<p>Theoretically,
<code>StabilizatioWindowSeconds = 1 hr - 10 mins = 50 mins (3000 secs)</code></p>
<p>Practically
Time taken for all Pods to terminate may vary depend on the scale down policy, your application etc...</p>
<p>So I set approx(according to my case)
<code>StabilizatioWindowSeconds = 1 hr - 20 mins = 40 mins (2400 secs)</code></p>
<p>Thus, your Scaled up pods can now alive for 40 mins, And starts terminating after 40 mins (In my case all pods terminated within max of 5 mins). So balance 15 mins for digital ocean to destroy the droplet.</p>
<p>CAUTION: Time calculated are depending on my use case and environment etc..</p>
<p>Add HPA behavior config for reference</p>
<pre><code>behavior:
    scaleDown:
      stabilizationWindowSeconds: 2400
      selectPolicy: Max
      policies:
      - type: percent
        value: 100
        periodSeconds: 15
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
      - type: Percent
        value: 100
        periodSeconds: 1
</code></pre>
