<p>Last night my Kubernetes cluster terimated 2 of my nodes and I can't figure out the details of what happened. </p>

<p><code>kubectl describe nodes</code> gives the following on the nodes that failed</p>

<pre><code>Conditions:
Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
----                 ------  -----------------                 ------------------                ------                       -------
NetworkUnavailable   False   Tue, 04 Sep 2018 21:57:00 +0000   Tue, 04 Sep 2018 21:57:00 +0000   RouteCreated                 RouteController created a route
OutOfDisk            False   Wed, 05 Sep 2018 12:12:33 +0000   Tue, 04 Sep 2018 21:56:27 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
MemoryPressure       False   Wed, 05 Sep 2018 12:12:33 +0000   Tue, 04 Sep 2018 21:56:27 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
DiskPressure         False   Wed, 05 Sep 2018 12:12:33 +0000   Tue, 04 Sep 2018 21:56:27 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
Ready                True    Wed, 05 Sep 2018 12:12:33 +0000   Tue, 04 Sep 2018 21:57:01 +0000   KubeletReady                 kubelet is posting ready status
</code></pre>

<p>So I know that <code>OutOfDisk, MemoryPressure and DiskPressure</code> were all in error state at some point last night, but what caused that to happen?</p>

<p>I also checked <code>kubectl get events --all-namespaces</code> and I get nothing.</p>

<p>Finally <code>kubectl describe pods</code> simply gave me this unhelpful information</p>

<pre><code>State:          Running
  Started:      Tue, 04 Sep 2018 22:03:47 +0000
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 30 Aug 2018 14:36:48 +0000
  Finished:     Tue, 04 Sep 2018 21:25:16 +0000
</code></pre>

<p>Is there a way to do a post mortem on this? I'd like to know more than that it was just out of disk space.</p>

Nate Bosscher
<p>I would reocommand you reading the following documentation: <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/</a></p>

<p>The first thought that came to my mind is examining the logs of your nodes/pods.</p>

<pre><code>kubectl logs
</code></pre>
