<p>I have a Spring Boot application running in a Docker container managed by Kubernetes whose job is to pull data from a queue, process it, and write back to a queue. Initially, the Deployment for this application is created with 1 replica. However, at some point I need to scale that Deployment up in order to process messages from the queue faster. Ideally, I'd like to have that existing 1 replica start processing messages, and then the new replicas can come up and help out.</p>
<p>The problem I've discovered is that when scaling the Deployment programmatically, the existing 1 replica is destroyed, and this typically is occurring in the middle of it processing some data, resulting in the output of that processing being lost.</p>
<p>Example of what I'd like to see happen:</p>
<ol>
<li>One replica processing message</li>
<li>Deployment gets scaled to 5 replicas</li>
<li>Four new replicas are spun up to help process messages leaving the existing.</li>
</ol>
<p>Is this at all possible? Thanks!</p>

printstacktrace
<p>Normally, Kubernetes scales up <strong>without</strong> restarting the existing Pods, so kind of unusual that this does not happen in your case.</p>
<p>That aside though, none of your Pods should shut down in a way that results in loss of messages. This suggests that you are <strong>missing code for graceful shutdown</strong>.</p>
<p>With Spring Boot and K8s you will have to do two things:</p>
<ul>
<li>Introduce a shutdown handler in Spring Boot that is triggered by K8s initial SIGTERM signal and makes sure the application does not accept new messages, but gets time to handle in-flight messages. Spring Boot provides <a href="https://www.baeldung.com/spring-shutdown-callbacks" rel="nofollow noreferrer">several mechanisms</a> for this.</li>
<li>Makes sure that K8s gives your application sufficient time for graceful shutdown by adequately configuring the <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podspec-v1-core" rel="nofollow noreferrer">terminationGracePeriodSeconds</a>. This is the time your applications gets to shut itself down in an orderly manner before K8s sends the SIGKILL signal.</li>
</ul>
