<p>I'm using the K3s distribution of Kubernetes which is deployed on
a Spot EC2 Instance in AWS.</p>
<p>I have scheduled a certain processing job and sometimes this job is being terminated and becomes in &quot;Unknown&quot; state (the job code is abnormally terminated)</p>
<pre><code>kubectl describe pod &lt;pod_name&gt;
</code></pre>
<p>it shows this:</p>
<pre><code> State:          Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Wed, 06 Jan 2021 21:13:29 +0000
      Finished:     Wed, 06 Jan 2021 23:33:46 +0000
</code></pre>
<p>The AWS logs show that the CPU consumption was 99% right before the crash.
From number of sources (<a href="https://jamesdefabia.github.io/docs/user-guide/pod-states/" rel="nofollow noreferrer">1</a>, <a href="https://github.com/kubernetes/kubernetes/issues/51333" rel="nofollow noreferrer">2</a>, <a href="https://www.reddit.com/r/kubernetes/comments/f7feec/pods_stuck_at_unknown_status_after_node_goes_down/" rel="nofollow noreferrer">3</a>) I saw that this can be a reason of a node crash but didn't see that one,
What may be the reason?</p>
<p>Thanks!</p>

sborpo
<p>The actual state of the Job is <code>Terminated</code> with the <code>Unknown</code> reason. In order to debug this situation you need to get a relevant logs from Pods created by your Job.</p>
<blockquote>
<p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup" rel="nofollow noreferrer">When a Job completes</a>, no more Pods are created, but the Pods are
not deleted either. Keeping them around allows you to still view the
logs of completed pods to check for errors, warnings, or other
diagnostic output.</p>
</blockquote>
<p>To do so, execute <code>kubectl describe job $JOB</code> to see the Pods' names under the Events section and than execute <code>kubectl logs $POD</code>.</p>
<p>If that won't be enough, you can try different ways to <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/" rel="nofollow noreferrer">Debug Pods</a>, such as:</p>
<ul>
<li><p>Debugging with container exec</p>
</li>
<li><p>Debugging with an ephemeral debug container, or</p>
</li>
<li><p>Debugging via a shell on the node</p>
</li>
</ul>
<p>The methods above will give you more info retarding the actual reasons behind the Job termination.</p>
