<p>I have a <code>replicaset</code> for <code>nginx-ingress</code> running in a cluster, with two instances. Two nights ago both containers were deleted at the same time (milliseconds apart from each other) and two new instances in the same replicaset were created. I'm clueless what triggered the delete. In the kubelet log I can see the following:</p>
<pre><code>kubelet[13317]: I0207 22:01:36.843804 13317 kubelet.go:1918] SyncLoop (DELETE, &quot;api&quot;): &quot;nginx-ingress-public-controller-6bf8d59c4c
</code></pre>
<p>And only a bit later in the log a failed liveness probe is listed:</p>
<pre><code>kubelet[13317]: I0207 22:01:42.596603 13317 prober.go:116] Liveness probe for &quot;nginx-ingress-public-controller-6bf8d59c4c (60c3f9e5-e228-44c8-abd5-b0a4a8507b5c):nginx-ingress-controller&quot; failed (failure): HTTP probe failed with statuscode: 500
</code></pre>
<p>In theory this might explain the pod deletion, but I'm confused by the order. Is this liveness probe failing because the delete command is already killing the underlying docker container, or is this what triggered the delete?</p>

<p>To determine what exactly caused deletion of your <code>nginx</code> pod is hard to guess without full logs. Also as you mention it's customer environment there might be many reasons. As I've asked in comments it might be <code>HPA</code> or <code>CA</code>, preemptible nodes, temporary network issues, etc.</p>
<p>Regarding the second part about pod deletion and <code>Liveness</code>, <code>Liveness</code> probe failed because <code>nginx</code> pod was in the <code>deletion</code> process.</p>
<p>One of <code>Kubernetes</code> default settings is <code>grace-period</code> equal to 30 seconds. In short it means that Pod will be in <code>Terminating</code> status for 30 seconds, and after this time it will be removed.</p>
<h2>Tests</h2>
<p>If you would like to verify it by yourself you can do some testing to confirm. It would require kubeadm master and change of <a href="https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#log-verbosity-level" rel="nofollow noreferrer">Verbosity</a>. You can do it by editing the <code>/var/lib/kubelet/kubeadm-flags.env</code> file (you must have root rights) and add <code>--v=X</code> where <code>X </code>is number <code>0-9</code>. Details which level shows specific logs can be found <a href="https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-output-verbosity-and-debugging" rel="nofollow noreferrer">here</a>.</p>
<ul>
<li>Set verbosity level at least to <code>level=5</code>, I've tested on <code>level=8</code></li>
<li>Deploy <code>Nginx Ingress Controller</code></li>
<li>Delete <code>Nginx Ingress Controller</code> pod manually</li>
<li>Check logs using <code>$ journalctl -u kubelet</code>, you can use <code>grep</code> to narrow output and save it to file
(<code>$ journalctl -u kubelet | grep ingress-nginx-controller-s2kfr &gt; nginx.log</code>)</li>
</ul>
<p>Below examples from my tests:</p>
<pre><code>#Liveness and Readiness probe works properly:
Feb 24 14:18:35 kubeadm kubelet[11922]: I0224 14:18:35.399156   11922 prober.go:126] Readiness probe for &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21):controller&quot; succeeded
Feb 24 14:18:40 kubeadm kubelet[11922]: I0224 14:18:40.587129   11922 prober.go:126] Liveness probe for &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21):controller&quot; succeeded

#Once Deletion process start you can find DELETE api and other information

Feb 24 14:18:46 kubeadm kubelet[11922]: I0224 14:18:46.900957   11922 kubelet.go:1931] SyncLoop (DELETE, &quot;api&quot;): &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21)&quot;
Feb 24 14:18:46 kubeadm kubelet[11922]: I0224 14:18:46.901057   11922 kubelet_pods.go:1482] Generating status for &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21)&quot;
Feb 24 14:18:46 kubeadm kubelet[11922]: I0224 14:18:46.901914   11922 round_trippers.go:422] GET https://10.154.15.225:6443/api/v1/namespaces/ingress-nginx/pods/ingress-nginx-controller-s2kfr
Feb 24 14:18:46 kubeadm kubelet[11922]: I0224 14:18:46.909123   11922 event.go:291] &quot;Event occurred&quot; object=&quot;ingress-nginx/ingress-nginx-controller-s2kfr&quot; kind=&quot;Pod&quot; apiVersion=&quot;v1&quot; type=&quot;Normal&quot; reason=&quot;Killing&quot; message=&quot;Stopping container controller&quot;

# This entry occurs as default grace-period-time was kept
Feb 24 14:18:46 kubeadm kubelet[11922]: I0224 14:18:46.947193   11922 kubelet_pods.go:952] Pod &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21)&quot; is terminated, but some containers are still running

# As Pod was in deletion, Probes failed.
Feb 24 14:18:50 kubeadm kubelet[11922]: I0224 14:18:50.584208   11922 prober.go:117] Liveness probe for &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21):controller&quot; failed (failure): HTTP probe failed with statuscode: 500
Feb 24 14:18:50 kubeadm kubelet[11922]: I0224 14:18:50.584338   11922 event.go:291] &quot;Event occurred&quot; object=&quot;ingress-nginx/ingress-nginx-controller-s2kfr&quot; kind=&quot;Pod&quot; apiVersion=&quot;v1&quot; type=&quot;Warning&quot; reason=&quot;Unhealthy&quot; message=&quot;Liveness probe failed: HTTP probe failed with statuscode: 500&quot;
Feb 24 14:18:52 kubeadm kubelet[11922]: I0224 14:18:52.045155   11922 kubelet_pods.go:952] Pod &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21)&quot; is terminated, but some containers are still running
Feb 24 14:18:55 kubeadm kubelet[11922]: I0224 14:18:55.398025   11922 prober.go:117] Readiness probe for &quot;ingress-nginx-controller-s2kfr_ingress-nginx(9046e404-1b9e-44fd-86f3-5a16ebf27c21):controller&quot; failed (failure): HTTP probe failed with statuscode: 500
</code></pre>
<p>In logs, the time between <code>SyncLoop (DELETE, &quot;api&quot;)</code> and <code>Liveness</code> probe is 4 seconds. In others, the test time was a few seconds (4-7 seconds difference).</p>
<p>If you would like to perform your own test you can change <code>Readiness</code> and <code>Liveness</code> probe check to 1 second (not 10 as is set by default) you would get probe issues in the same second as <code>Delete api</code>.</p>
<pre><code>Feb 24 15:09:40 kubeadm kubelet[11922]: I0224 15:09:40.865718   11922 prober.go:126] Liveness probe for &quot;ingress-nginx-controller-wwrdw_ingress-nginx(427bc9d6-261e-4427-b034-7abe8cbbfea6):controller&quot; succeeded
Feb 24 15:09:41 kubeadm kubelet[11922]: I0224 15:09:41.488819   11922 kubelet.go:1931] SyncLoop (DELETE, &quot;api&quot;): &quot;ingress-nginx-controller-wwrdw_ingress-nginx(427bc9d6-261e-4427-b034-7abe8cbbfea6)&quot;
...
Feb 24 15:09:41 kubeadm kubelet[11922]: I0224 15:09:41.865422   11922 prober.go:117] Liveness probe for &quot;ingress-nginx-controller-wwrdw_ingress-nginx(427bc9d6-261e-4427-b034-7abe8cbbfea6):controller&quot; failed (failure): HTTP probe failed with statuscode: 500
</code></pre>
<p>Good explanation of <code>syncLoop</code> you can find in <a href="https://www.alibabacloud.com/blog/understanding-the-kubelet-core-execution-frame_593904" rel="nofollow noreferrer">Alibaba docs</a></p>
<blockquote>
<p>As indicated in the comments, the <code>syncLoop</code> function is the major cycle of <code>Kubelet</code>. This function listens on the updates, obtains the latest <code>Pod</code> configurations, and <code>synchronizes</code> the running state and desired state. In this way, all <code>Pods</code> on the local node can run in the expected states. Actually, <code>syncLoop</code> only encapsulates <code>syncLoopIteration</code>, while the <code>synchronization</code> operation is carried out by <code>syncLoopIteration</code>.</p>
</blockquote>
<h2>Conclusion</h2>
<p>If you don't have additional logging to save outputs from pods before termination it's hard to determine the root cause after a while since that event.</p>
<p>In the setup you have provided, the <code>Liveness</code> probe failed because <code>nginx-ingress</code> pod was already in the termination process. <code>Liveness probe</code> fail did not trigger pod deletion but it was the result of that deletion.</p>
<p>In addition, you can also check <a href="https://github.com/kubernetes/kubernetes/blob/165e5664b0e65d2e0f8fbcfec16aa95f2bc3cb19/pkg/kubelet/kubelet.go" rel="nofollow noreferrer">Kubelet</a> and <a href="https://github.com/kubernetes/kubernetes/blob/165e5664b0e65d2e0f8fbcfec16aa95f2bc3cb19/pkg/kubelet/prober/prober.go" rel="nofollow noreferrer">Prober</a> source code.</p>
