<p>In my K8S workloads, I implement Readiness probe and Liveness probe for pods health check.</p>

<p>I'm wondering that should I set the interval (<code>periodSeconds</code>) as low as 1 sec, as it will consume more resources, right? </p>

<p>Is there best practices when doing the pod health check?</p>

<p>Firstly, it is important to understand the difference between Liveness and Readiness. The tl;dr is: Liveness is about whether K8s should kill and restart the container, Readiness is about whether the container is able to accept requests. It is likely that you want different parameters for both.</p>

<p>Whether K8s takes any action based on the outcome of the probe depends on the <code>failureThreshold</code>. This is the number of times in a row the probe has to fail before K8s does something. If you combine this with <code>periodSeconds</code> you can tune the sensitivity of your probes.</p>

<p>In general you want to balance:</p>

<ul>
<li>the time it takes K8s to take action with how quickly your service can be expected to recover based on the probe</li>
<li>the "cost" of the probes. For example if your Readiness probe connects to a database, then you are adding 1 Query Per Second (QPS) load to your database per replica (With 100 replicas, you would be generating 100QPS just through probes!)</li>
<li>the reliability of your probe, also known as "flakiness". What is the false negative rate - i.e what proportion of the time the probe reports failed but the service is actually running with in expected performance rates</li>
</ul>

<p>Here is one way of thinking about it:</p>

<ul>
<li>Work out how long your service can be in the failed state before K8s should take action. This should be based on how long it would take to recover (e.g. restart in the case of Liveness)</li>
<li>If a probe is "expensive", have a longer <code>periodSeconds</code> and smaller <code>failureThreshold</code></li>
<li>If a probe is "flaky" (i.e. occasionally reports failed and then reports working very quickly afterwards) have a shorter <code>periodSeconds</code> and larger <code>failureThreashold</code>.</li>
</ul>
