<p>Scenario: A K8S pod has more than one container and liveness/readiness probes are configured for each of the containers. Now if the liveness probe is succeeding on some containers and failing on few containers, what will k8s do.</p>
<ol>
<li>will it restart only the failing containers<br />
OR</li>
<li>will it restart the entire pod.</li>
</ol>

samshers
<blockquote>
<p>if the liveness probe is succeeding on some containers and failing on few containers, what will k8s do?</p>
</blockquote>
<p><strong>It will restart only the failing containers.</strong></p>
<p>In <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes" rel="noreferrer">Pod Lifecycle - Container Probes</a> you have listed all 3 probes: <code>liviness</code>, <code>readiness</code> and <code>startup</code>.</p>
<blockquote>
<p><code>livenessProbe</code>: Indicates whether the <code>container</code> is running. <strong>If the liveness probe fails, the kubelet kills the container</strong>, and the container is subjected to its <code>restart policy</code>. If a Container does not provide a liveness probe, the default state is Success.</p>
</blockquote>
<p>In <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-command" rel="noreferrer">Configure Liveness, Readiness and Startup Probes - Define a liveness command</a> you have example and it's mentioned that:</p>
<blockquote>
<p>If the command succeeds, it returns 0, and the kubelet considers the container to be alive and healthy. If the command returns a non-zero value, the kubelet <strong>kills the container and restarts it</strong>.</p>
</blockquote>
<p>The same situation is in case <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request" rel="noreferrer">HTTP request liveness probe</a>:</p>
<blockquote>
<p>If the handler for the server's /healthz path returns a success code, the kubelet considers the container to be alive and healthy. If the handler returns a failure code, the kubelet kills the container and restarts it.</p>
</blockquote>
<p>And with <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-tcp-liveness-probe" rel="noreferrer">TCP liveness probe</a>:</p>
<blockquote>
<p>The kubelet will run the first liveness probe 15 seconds after the container starts. Just like the readiness probe, this will attempt to connect to the goproxy container on port 8080. If the liveness probe fails, the container will be restarted.</p>
</blockquote>
<h2>Tests</h2>
<p>If you would like create own test you can use this example of HTTP Liveness probe:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http-probe
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: X-Custom-Header
          value: Awesome
      initialDelaySeconds: 0
      periodSeconds: 5      
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: X-Custom-Header
          value: Awesome
      initialDelaySeconds: 5
      periodSeconds: 10     
      successThreshold: 1
      failureThreshold: 3 
  - name: nginx
    image: nginx
</code></pre>
<p>After a while you will be able to see that the <code>container</code> was restarted, <code>restart count</code> was increased, but the <code>pod</code> is still existing as <code>Age</code> is still counting.</p>
<pre><code>$ kubectl get po -w
NAME                  READY   STATUS    RESTARTS   AGE
liveness-http-probe   2/2     Running   0          20s
liveness-http-probe   1/2     Running   0          23s
liveness-http-probe   1/2     Running   1          42s
liveness-http-probe   2/2     Running   1          43s
liveness-http-probe   1/2     Running   1          63s
...
liveness-http-probe   1/2     Running   5          3m23s
liveness-http-probe   2/2     Running   5          3m23s
liveness-http-probe   1/2     Running   5          3m43s
liveness-http-probe   1/2     CrashLoopBackOff   5          4m1s
liveness-http-probe   1/2     Running            6          5m25s
liveness-http-probe   2/2     Running            6          5m28s
liveness-http-probe   1/2     Running            6          5m48s
liveness-http-probe   1/2     CrashLoopBackOff   6          6m2s
liveness-http-probe   1/2     Running            7          8m46s
liveness-http-probe   2/2     Running            7          8m48s
...
liveness-http-probe   2/2     Running   11         21m
liveness-http-probe   1/2     Running   11         21m
liveness-http-probe   1/2     CrashLoopBackOff   11         22m
liveness-http-probe   1/2     Running            12         27m
...
liveness-http-probe   1/2     Running            13         28m
liveness-http-probe   1/2     CrashLoopBackOff   13         28m
</code></pre>
<p>And in the pod description you will see duplicate <code>Warnings</code> like <code>(x8 over 28m)</code>, <code>(x84 over 24m)</code> or <code>(x2 over 28m)</code>.</p>
<pre><code>  Normal   Pulling    28m (x2 over 28m)     kubelet            Pulling image &quot;k8s.gcr.io/liveness&quot;
  Normal   Killing    28m                   kubelet            Container liveness failed liveness probe, will be restarted
  Normal   Started    28m (x2 over 28m)     kubelet            Started container liveness
  Normal   Created    28m (x2 over 28m)     kubelet            Created container liveness
  Normal   Pulled     28m                   kubelet            Successfully pulled image &quot;k8s.gcr.io/liveness&quot; in 561.418121ms
  Warning  Unhealthy  27m (x8 over 28m)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  27m (x4 over 28m)     kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Normal   Pulled     13m (x2 over 14m)     kubelet            (combined from similar events): Successfully pulled image &quot;k8s.gcr.io/liveness&quot; in 508.892628ms
  Warning  BackOff    3m45s (x84 over 24m)  kubelet            Back-off restarting failed container
</code></pre>
<p>Lately I did some tests with <code>liveness</code> and <code>readiness</code> probes in thread - <a href="https://stackoverflow.com/questions/66299478/liveness-probe-readiness-probe-not-called-in-expected-duration/66552156">Liveness Probe, Readiness Probe not called in expected duration</a>. It can provide you additional information.</p>
