<p>When the pod is Evicted by disk issue, I found there are two reasons:</p>
<ol>
<li>The node had condition: <code>[DiskPressure]</code></li>
<li>The node was low on resource: ephemeral-storage. Container NAME was using 16658224Ki, which exceeds its request of 0.</li>
</ol>
<p>I found <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions" rel="nofollow noreferrer">Node conditions</a> for <code>DiskPressure</code>.</p>
<p>What is the difference?</p>

<p>Both reasons are caused by the same error - that the worker node has ran out of disk space. However, the difference is when they are exactly happening.</p>
<p>To answer this question I decided to dig inside <a href="https://github.com/kubernetes/kubernetes" rel="noreferrer">Kubernetes source code</a>.</p>
<p>Starting with <code>The node had condition: [DiskPressure]</code> error.</p>
<p>We can find that it is used in <code>pkg/kubelet/eviction/helpers.go</code> file, <a href="https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/eviction/helpers.go#L44" rel="noreferrer">line 44</a>:</p>
<pre><code>nodeConditionMessageFmt = &quot;The node had condition: %v. &quot;
</code></pre>
<p>This variable is used by <code>Admit</code> function in <code>pkg/kubelet/eviction/eviction_manager.go</code> file, <a href="https://github.com/kubernetes/kubernetes/blob/2ba872513da76568274885f8c5736b415a77b0cd/pkg/kubelet/eviction/eviction_manager.go#L137" rel="noreferrer">line 137</a>.</p>
<p><code>Admit</code> function is used by <code>canAdmitPod</code> function in <code>pkg/kubelet/kubelet.go</code> file,  <a href="https://github.com/kubernetes/kubernetes/blob/047a6b9f861b2cc9dd2eea77da752ac398e7546f/pkg/kubelet/kubelet.go#L1932" rel="noreferrer">line 1932</a>:</p>
<p><code>canAdmitPod</code> function is used by <code>HandlePodAdditions</code> function, also in the <code>kubelet.go</code> file, <a href="https://github.com/kubernetes/kubernetes/blob/047a6b9f861b2cc9dd2eea77da752ac398e7546f/pkg/kubelet/kubelet.go#L2195" rel="noreferrer">line 2195</a>.</p>
<p>Comments in the code in <code>Admit</code> and <code>canAdmitPod</code> functions:</p>
<blockquote>
<p>canAdmitPod determines if a pod can be admitted, and gives a reason if it cannot. &quot;pod&quot; is new pod, while &quot;pods&quot; are all admitted pods.
The function returns a boolean value indicating whether the pod can be admitted, a brief single-word reason and a message explaining why the pod cannot be admitted.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>Check if we can admit the pod; if not, reject it.</p>
</blockquote>
<p>So based on this analysis we can conclude that <code>The node had condition: [DiskPressure]</code> error message happens when kubelet agent won't admit <strong>new</strong> pods on the node, that means they won't start.</p>
<hr />
<p>Now moving on to the second error - <code>The node was low on resource: ephemeral-storage. Container NAME was using 16658224Ki, which exceeds its request of 0</code></p>
<p>Similar as before, we can find it in <code>pkg/kubelet/eviction/helpers.go</code> file, <a href="https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/eviction/helpers.go#L42" rel="noreferrer">line 42</a>:</p>
<pre><code>nodeLowMessageFmt  =  &quot;The node was low on resource: %v. &quot;
</code></pre>
<p>This variable is used in the same file by <code>evictionMessage</code> function, <a href="https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/eviction/helpers.go#L1003" rel="noreferrer">line 1003</a>.</p>
<p><code>evictionMessage</code> function is used by <code>synchronize</code> function in <code>pkg/kubelet/eviction/eviction_manager.go</code> file, <a href="https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/eviction/eviction_manager.go#L231" rel="noreferrer">line 231</a></p>
<p><code>synchronize</code> function is used by <code>start</code> function, in the same file, <a href="https://github.com/kubernetes/kubernetes/blob/2ba872513da76568274885f8c5736b415a77b0cd/pkg/kubelet/eviction/eviction_manager.go#L177" rel="noreferrer">line 177</a>.</p>
<p>Comments in the code in the <code>synchronize</code> and <code>start</code> functions:</p>
<blockquote>
<p>synchronize is the main control loop that enforces eviction thresholds.
Returns the pod that was killed, or nil if no pod was killed.</p>
</blockquote>
<blockquote>
<p>Start starts the control loop to observe and response to low compute resources.</p>
</blockquote>
<p>So we can conduct that the error <code>The node was low on resource:</code> error message happens when kubelet agent decides to kill currently <strong>running</strong> pods on the node.</p>
<hr />
<p>It is worth emphasising that both error messages comes from node conditions (which are set in function <code>synchronize</code>, <a href="https://github.com/kubernetes/kubernetes/blob/2ba872513da76568274885f8c5736b415a77b0cd/pkg/kubelet/eviction/eviction_manager.go#L308" rel="noreferrer">line 308</a>, to the values detected by eviction manager). Then kubelet's agent makes the decisions that results in these two error messages.</p>
<hr />
<p><strong>To sum up</strong>:</p>
<p>Both errors are due to insufficient disk space, but:</p>
<ul>
<li><code>The node had condition:</code> error is related to the pods that are <strong>about to start</strong> on the node, but they can't</li>
<li><code>The node was low on resource:</code> error is related to the <strong>currently running pods</strong> that must be terminated</li>
</ul>
