<p>I've got application with 10 pods and traffic is load balanced between all pods. There was an issue that caused transactions queued up and few pods could not recover properly or took a long time to process the queue once the issue was fixed. The new traffic was still too much for some of the pods.</p>
<p>I'm wondering if I can block new traffic to particular pod(s) in a replicaset and let them process the queue and once the queue is processed then let the new traffic come in again?</p>

<p>For that you can use the probe to handle this scenario</p>
<p>A <strong><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes" rel="noreferrer">Readiness</a></strong> probe is one way to do it.</p>
<p>What <strong>probes</strong> to do is, continuously check inside the container or POD for the process is up or not on configured time interval.</p>
<p>Example</p>
<pre><code>readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5
</code></pre>
<p>you can create the endpoint into the application which will be checked by the K8s automatically and if K8s will <strong>200</strong> it will mark the POD as <strong>Ready</strong> to handle the traffic. Or else mark as <strong>Unready</strong> not to handle traffic.</p>
<p><strong>Note</strong> :</p>
<p>Readiness and liveness probes can be used in parallel for the same container. Using both can ensure that traffic does not reach a container that is not ready for it, and that containers are restarted when they fail.</p>
<p>The Readiness probe won't restart your POD if it's failing, while the liveness probe will restart your POD or container if it's failing and sending <strong>400</strong>.</p>
<p>In your scenario, it's better to use the <strong>Readiness</strong> probe, so the process keeps running and never gets restarted. Once application ready to handle traffic K8s will get the <strong>200</strong> responses on endpoint.</p>
<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/" rel="noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/</a></p>
