<p>Often times, when I want to check out what's wrong with <code>Pod</code>s that go to a state of <code>CrashLoopBackoff</code> or <code>Error</code>, I do the following. I change the pod command to <code>sleep 10000</code> and run <code>kubectl exec -ti POD_NAME bash</code> in my terminal to further inspect the environment and code. The problem is that it terminates very soon and without exception. It has been quite annoying to inspect the content of my pod. </p>

<p><strong>My config</strong></p>

<p>The result of <code>kubectl version</code>:</p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-15T15:50:38Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>The result of <code>helm version</code>:</p>

<pre><code>version.BuildInfo{Version:"v3.0.1", GitCommit:"7c22ef9ce89e0ebeb7125ba2ebf7d421f3e82ffa", GitTreeState:"clean", GoVersion:"go1.13.4"}
</code></pre>

<p>OS: <code>MacOS Catalina 10.15.2</code></p>

<p><code>Docker</code> version: <code>19.03.5</code></p>

<p>I run my stuff using <code>helm</code> and <code>helmfile</code>, and my releases usually include a <code>Deployment</code> and a <code>Service</code>.</p>

<p>Let me know if any additional info can help.</p>

<p>Any help is appreciated! </p>

mhyousefi
<p>Try to install Golang in version <strong>1.13.4+</strong>. You have <strong>go1.12.12</strong> version of kubectl server which casues a lot of problems with compatibility. So you have to update it. If you are upgrading from an older version of Go you must first remove the existing version.
Take a look here: <a href="https://gist.github.com/nikhita/432436d570b89cab172dcf2894465753" rel="nofollow noreferrer">upgrading-golang</a>.</p>

<p>Apply changes in your pod definition file, add following lines under container definition:</p>

<pre><code>#Just spin &amp; wait forever
command: [ "/bin/bash", "-c", "--" ]
args: [ "trap : TERM INT; sleep infinity &amp; wait" ]
</code></pre>

<p>This will keep your container alive until it is told to stop. Using trap and wait will make your container react immediately to a stop request. Without trap/wait stopping will take a few seconds.</p>

<p>If you think it is networking problem use a <a href="https://danielmiessler.com/study/tcpdump/" rel="nofollow noreferrer">tcpdump</a>.</p>

<p>Tcpdump is a tool to that captures network traffic and helps you troubleshoot some common networking problems. Here is a quick way to capture traffic on the host to the target container with IP 172.28.21.3.</p>

<p>We are going to join the one container and will be trying to reach out another container:</p>

<pre><code>kubectl exec -ti testbox-2460950909-5wdr4 -- /bin/bash
$ curl http://ip:port
</code></pre>

<p>On the host with a container we are going to capture traffic related to container target IP:</p>

<pre><code>$ tcpdump -i any host ip

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:15:59.903566 IP 172.28.128.132.60358 &gt; 172.28.21.3.5000: Flags [S], seq 3042274422, win 28200, options [mss 1410,sackOK,TS val 10056152 ecr 0,nop,wscale 7], length 0
20:15:59.903566 IP 172.28.128.132.60358 &gt; 172.28.21.3.5000: Flags [S], seq 3042274422, win 28200, options [mss 1410,sackOK,TS val 10056152 ecr 0,nop,wscale 7], length 0
20:15:59.905481 ARP, Request who-has 172.28.21.3 tell 10.244.27.0, length 28
20:16:00.907463 ARP, Request who-has 172.28.21.3 tell 10.244.27.0, length 28
20:16:01.909440 ARP, Request who-has 172.28.21.3 tell 10.244.27.0, length 28
20:16:02.911774 IP 172.28.128.132.60358 &gt; 172.28.21.3.5000: Flags [S], seq 3042274422, win 28200, options [mss 1410,sackOK,TS val 10059160 ecr 0,nop,wscale 7], length 0
20:16:02.911774 IP 172.28.128.132.60358 &gt; 172.28.21.3.5000: Flags [S], seq 3042274422, win 28200, options [mss 1410,sackOK,TS val 10059160 ecr 0,nop,wscale 7], length 0
As you see there is a trouble on the wire as kernel fails to route the packets to the target IP.
</code></pre>

<p>You can also debug pod using <code>kubectl logs</code> command:</p>

<p>Running <code>kubectl logs -p</code> will fetch logs from existing resources at API level. This means that terminated pods' logs will be unavailable using this command.</p>

<p>The best way is to have your logs centralized via <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures" rel="nofollow noreferrer">logging agents or directly pushing these logs into an external service</a>.</p>

<p>Alternatively and given the logging architecture in Kubernetes, you might be able to fetch the logs directly from the log-rotate files in the node hosting the pods. However, this option might depend on the Kubernetes implementation as log files might be deleted when the pod eviction is triggered.</p>

<p>Take a look here: <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#my-pod-is-crashing-or-otherwise-unhealthy" rel="nofollow noreferrer">pod-debugging</a>.</p>

<p>Take a look on official documentation: <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/" rel="nofollow noreferrer">kubectl-exec</a>.</p>
