<p>What happens when Kubernetes <code>liveness-probe</code> returns false?
Does Kubernetes restart that pod immediately?</p>

J.J. Beam
<p>First, please note that <code>livenessProbe</code> concerns <strong>containers</strong> in the pod, not the pod itself. So if you have multiple containers in one pod, only the affected container will be restarted.</p>
<p>It's worth noting, that there is <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="noreferrer">parameter <code>failureThreshold</code></a>, which is set by default to 3. So, after 3 failed probes a container will be restarted:</p>
<blockquote>
<p><code>failureThreshold</code>: When a probe fails, Kubernetes will try  <code>failureThreshold</code>  times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready. Defaults to 3. Minimum value is 1.</p>
</blockquote>
<p>Ok, we have information that a container is restarted after 3 failed probes - but what does it mean to <em>restart</em>?</p>
<p>I found a good article about <em>how</em> Kubernetes terminates a pods - <a href="https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace" rel="noreferrer">Kubernetes best practices: terminating with grace</a>. Seems for container restart caused by liveness probe it's similar - I will share my experience below.</p>
<p>So basically when container is being terminated by liveness probe steps are:</p>
<ul>
<li>if there is a <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" rel="noreferrer"><code>PreStop</code> hook</a>, it will be executed</li>
<li><a href="https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html" rel="noreferrer">SIGTERM signal</a> is sent to the container</li>
<li>Kubernetes waits for grace period</li>
<li>After grace period, <a href="https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html" rel="noreferrer">SIGKILL signal</a> is sent to a pod</li>
</ul>
<p>So... if an app in your container is catching the SIGTERM signal properly, then the container will shut-down and will be started again. Typically it's happening pretty fast (as I tested for the NGINX image) - almost immediately.</p>
<p>Situation is different when SIGTERM is not supported by your application. It means after <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="noreferrer"><code>terminationGracePeriodSeconds</code> period</a> the SIGKILL signal is sent, it means the container will be forcibly removed.</p>
<p>Example below (modified example from <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-command" rel="noreferrer">this doc</a>) + I set <code>failureThreshold: 1</code></p>
<p>I have following pod definition:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: nginx
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      periodSeconds: 10
      failureThreshold: 1
</code></pre>
<p>Of course there is no <code>/tmp/healthy</code> file, so livenessProbe will fail. The NGINX image is properly catching the SIGTERM signal, so the container will be restarted almost immediately (for every failed probe). Let's check it:</p>
<pre><code>user@shell:~/liveness-test-short $ kubectl get pods
NAME                                   READY   STATUS             RESTARTS   AGE
liveness-exec                          0/1     CrashLoopBackOff   3          36s
</code></pre>
<p>So after ~30 sec the container is already restarted a few times and it's status is <a href="https://sysdig.com/blog/debug-kubernetes-crashloopbackoff/" rel="noreferrer">CrashLoopBackOff</a> as expected. I created the same pod without livenessProbe and I measured the time need to shutdown it:</p>
<pre><code>user@shell:~/liveness-test-short $ time kubectl delete pod liveness-exec
pod &quot;liveness-exec&quot; deleted

real    0m1.474s
</code></pre>
<p>So it's pretty fast.</p>
<p>The similar example but I added <code>sleep 3000</code> command:</p>
<pre class="lang-yaml prettyprint-override"><code>...
image: nginx
    args:
    - /bin/sh
    - -c
    - sleep 3000
...
</code></pre>
<p>Let's apply it and check...</p>
<pre><code>user@shell:~/liveness-test-short $ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
liveness-exec                          1/1     Running   5          3m37s
</code></pre>
<p>So after ~4 min there are only 5 restarts. Why? Because we need to wait for full <code>terminationGracePeriodSeconds</code> period (default is 30 seconds) for every restart. Let's measure time needed to shutdown:</p>
<pre><code>user@shell:~/liveness-test-short $ time kubectl delete pod liveness-exec
pod &quot;liveness-exec&quot; deleted

real    0m42.418s
</code></pre>
<p>It's much longer.</p>
<p><strong>To sum up:</strong></p>
<blockquote>
<p>What happens when Kubernetes liveness-probe return false? Does Kubernetes restart that pod immediately?</p>
</blockquote>
<p>The short answer is:  by default no. Why?</p>
<ul>
<li>Kubernetes will restart a container in a pod after <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="noreferrer"><code>failureThreshold</code></a> times. By default it is 3 times - so after 3 failed probes.</li>
<li>Depends on your configuration of the container, time needed for container termination could be very differential</li>
<li>You can adjust both <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes" rel="noreferrer"><code>failureThreshold</code></a> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="noreferrer"><code>terminationGracePeriodSeconds</code> period</a> parameters, so the container will be restarted immediately after every failed probe.</li>
</ul>
