<p>Our worker service is a long running service. When scale in happen or deployment happen, we expect the PODs can finish existing works (max alive 1 week) then exit.</p>

<p>What I have tried is I make a deployment with 10 pods and set terminationgraceperiodseconds = 604800, and then scale down instance to 1, that works good.</p>

<p><strong>Question</strong> here is our service will have <strong>hundreds</strong> POD, which means worse case is hundreds pod will be in terminating status, run 7 days then exit. Is this workable in K8s world, or any potential issue?
Thank you if any comment~</p>

Gu111gu
<p>Google starts and destroys over <a href="https://cloud.google.com/containers" rel="nofollow noreferrer">7 billion pods</a> per week. This is the purpose they were made for. As long as you are persisting the necessary data to disk, Kubernetes will replicate the state of the pod exactly as you have configured. A <a href="https://research.google/pubs/pub44843/" rel="nofollow noreferrer">paper</a> has also been published on how this scale may be achieved.</p>
