<p>I am quite confused about readiness probe. Suppose I use httpGet with /health as the probing endpoint. Once the readiness check returns 500, the server will stop serving traffic. Then how can the /health endpoint work? In other words, once a readiness check fails, how can it ever work again since it can no longer answer to future /health checks?</p>
<p>I guess one valid explanation is that the path is invoked locally? (i.e. not through the https:${ip and port}/health)</p>

<p>You have typo.. you said :</p>
<blockquote>
<p>Once the readiness check returns 500, the <strong>server</strong> will stop serving traffic.</p>
</blockquote>
<p>However, it should be :</p>
<blockquote>
<p>Once the readiness check returns 500, the <strong>k8s service</strong> will stop serving traffic.</p>
</blockquote>
<p><a href="https://i.stack.imgur.com/RZqkN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RZqkN.png" alt="enter image description here" /></a></p>
<p>k8s service behaves like a load balancer for multi-pods.</p>
<ul>
<li>If pod is ready, an endpoint will be created for the ready pod, and the traffic will be received.</li>
<li>If pod is not ready, its endpoint will be removed and it will not more receive traffic.</li>
</ul>
<p>While <strong>Readiness Probe</strong> decides to forward traffic or not, <strong>Liveness Probe</strong> decides to restart the Pod or not.</p>
<p>If you want to get rid off unhealthy Pod, you have to specify also <strong>Liveness Probe</strong>.</p>
<h2>So let's summarize:</h2>
<p>To get full HA deployment you need 3 things together:</p>
<ul>
<li>Pod are managed by <strong>Deployment</strong> which will maintain a number of replicas.</li>
<li><strong>Liveness Probe</strong> will help to remove/restart the unlheathy pod.. After somtime ( 6 restarts), the Pod will be unhealthy and the Deployment will take care to bring new one.</li>
<li><strong>Readiness Probe</strong> will help forward traffic to only ready pods : Either at beginning of run, or at the end of run ( graceful shutdown).</li>
</ul>
