<p>I'm unable to find any references other than this link that confirms that the failure has to be consecutive. <a href="https://github.com/kubernetes/website/issues/37414" rel="nofollow noreferrer">https://github.com/kubernetes/website/issues/37414</a></p>
<p>Background: Our Java application is getting restarted every day because of liveness probe failure. The application's access logs don't show 3 consecutive failures. So wanted to understand the behavior of probes.</p>

<p>Liveness check is created when Kubernetes creates pod and is recreated each time that Pod is restarted. In your configuration you have set initialDelaySeconds: 20 so after creating a pod, Kubernetes will wait 20 seconds, then it will call liveness probe 3 times (as default value failureThreshold: 3). After 3 fails, Kubernetes will restart this pod according to RestartPolicy. Also in logs you will be able to find in logs.</p>
<p>When you are using  <code>kubectl get events</code> you are getting events only from the last hour.</p>
<pre><code>Kubectl get events

LAST SEEN           TYPE      REASON                    OBJECT             
47m             Normal    Starting                  node/kubeadm      
43m                 Normal    Scheduled                 pod/liveness-http   
43m             Normal    Pulling                   pod/liveness-http   
43m                 Normal    Pulled                    pod/liveness-http 
43m                 Normal    Created                   pod/liveness-http   
43m             Normal    Started                   pod/liveness-http   
4m41s           Warning   Unhealthy                 pod/liveness-http 
40m                 Warning   Unhealthy                 pod/liveness-http   
12m20s              Warning   BackOff                   pod/liveness-http   
</code></pre>
<p>same command after ~1 hour:</p>
<pre><code>LAST SEEN       TYPE         REASON         OBJECT            
43s             Normal      Pulling          pod/liveness-http   
8m40s               Warning     Unhealthy       pod/liveness-http  
20m                 Warning     BackOff         pod/liveness-http 
</code></pre>
<p>So that might be the reason you are seeing only one failure.</p>
<p>Liveness probe can be configured using the fields below:</p>
<ul>
<li><p>initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.</p>
</li>
<li><p>periodSeconds: How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.</p>
</li>
<li><p>timeoutSeconds: Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1.</p>
</li>
<li><p>successThreshold: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness. Minimum value is 1.</p>
</li>
<li><p>failureThreshold: When a probe fails, Kubernetes will try failureThreshold times before giving up. Giving up in case of liveness probe means restarting the container. In case of a readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.</p>
</li>
</ul>
<p>If you set the minimal values for periodSeconds, timeoutSeconds, successThreshold and failureThreshold you can expect more frequent checks and faster restarts.</p>
<p>Liveness probe :</p>
<ul>
<li>Kubernetes will restart a container in a pod after <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="nofollow noreferrer">failureThreshold</a> times. By default it is 3 times - so after 3 failed probes.</li>
<li>Depending on your configuration of the container, time needed for container termination could be very differential</li>
<li>You can adjust both <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="nofollow noreferrer">failureThreshold</a> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">terminationGracePeriodSeconds</a> period parameters, so the container will be restarted immediately after every failed probe</li>
</ul>
<p>In <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/" rel="nofollow noreferrer">liveness probe configuration</a> and <a href="https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes" rel="nofollow noreferrer">best practices</a> you can find more information.</p>
