<p>Today my kubernetes cluster(v1.15.2) node disk full and cause pods give this tips:</p>

<pre><code>Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.261578065: no space left on device, which is unexpected.
MountVolume.SetUp failed for volume "default-token-xnrwt" : mkdir /opt/k8s/k8s/kubelet/pods/67eaa71b-adf4-4365-a1c7-42045d5e9426: no space left on device
</code></pre>

<p>I login into server and find the disk usage is 100%, so I remove some log file and release 10GB + disk space, but now it seems pod is not recovery automaticlly and still have this error tips:</p>

<p><a href="https://i.stack.imgur.com/bTlcL.png" rel="noreferrer"><img src="https://i.stack.imgur.com/bTlcL.png" alt="enter image description here"></a></p>

<p>what should I do to fix this problem? I am try to restart all pods, and all pods works fine. But finally I found the error tips message still give me tips no space left and did not disappear automaticlly. I check the node status and find the node has no disk pressure. How to make the error tips disappear?</p>

<p>Other possibility is incorrect unit values for resource requests/limits (ex, using <code>mi</code> instead of <code>Mi</code>).</p>
<p>For example:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
spec:
  containers:
    - name: {container_name}
      resources:
        limits:
          memory: &quot;512mi&quot; # incorrect; should be &quot;512Mi&quot;
          cpu: &quot;200m&quot;
</code></pre>
