<p>Anyone experienced this <strong>UnexpectedAdmissionError</strong> :
<em>Update plugin resources failed due to failed to write checkpoint file &quot;kubelet_internal_checkpoint&quot;: mkdir /var: file exists, which is unexpected.</em></p>
<p><em>$ kubectl get pods</em></p>
<pre><code>NAME                    READY   STATUS                     RESTARTS   AGE
test-6b5ddf5dd4-22tlr   0/1     UnexpectedAdmissionError   0          9m
</code></pre>
<p>This pod is from a deployment</p>
<p><em>$ kubectl describe pod test-6b5ddf5dd4-22tlr</em></p>
<pre><code>Name:           test-6b5ddf5dd4-22tlr 
Priority:       0
Node:           node-1
Start Time:     Mon, 07 Dec 2020 18:37:46 +0000
Labels:         pod-template-hash=6b5ddf5dd4
Annotations:    kubernetes.io/psp: 99-restricted
                seccomp.security.alpha.kubernetes.io/pod: docker/default
Status:         Failed
Reason:         UnexpectedAdmissionError
Message:        Pod Update plugin resources failed due to failed to write checkpoint file &quot;kubelet_internal_checkpoint&quot;: mkdir /var: file exists, which is unexpected.
IP:
IPs:            &lt;none&gt;
Controlled By:  ReplicaSet/test-6b5ddf5dd4
Containers:
  test-pod:
    Image:        xxxx
    Port:         8070/TCP
    Host Port:    0/TCP
    Environment:  &lt;none&gt;
    Mounts:
      /app/config/ from application-config (ro)
      /app/data_vol from datavolume (rw)
      /app/logback/ from logback-config (ro)
      /app/logs_vol from logvolume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kbwnx (ro)
Volumes:
  logvolume:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    server1
    Path:      /release
    ReadOnly:  false
  datavolume:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    server2
    Path:      /ivol/
    ReadOnly:  false
  application-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      configmap
    Optional:  false
  logback-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      logback-config-map
    Optional:  false
  default-token-kbwnx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kbwnx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                    Age   From                 Message
  ----     ------                    ----  ----                 -------
  Normal   Scheduled                 28m   default-scheduler    Successfully assigned test-6b5ddf5dd4-22tlr to node-1
  Warning  UnexpectedAdmissionError  28m   kubelet, node-1  Update plugin resources failed due to failed to write checkpoint file &quot;kubelet_internal_checkpoint&quot;: mkdir /var: file exists, which is unexpected.
</code></pre>

Chakra
<p>Posting this as <code>Community Wiki</code> for better visibility.</p>
<p><strong>Solution</strong> of this issue was <strong>rebooting affected node.</strong></p>
<p>OP confirmed that issue was with only one node, as other pods on other nodes were not affected. As this issue cannot be replicated and now it's impossible to determine root cause, please find some guidelines and troubleshooting documents below which might help resolve similar issue in the future.</p>
<p>It's best to start with the most general troubleshooting like <code>$ kubectl describe node &lt;nodename&gt;</code> tog et overall information about node (Capacity, Allocation, Conditions, Events, etc.).</p>
<p>Later it's good to check <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">Resources Quota</a> and verify if node have enough resources - <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/" rel="nofollow noreferrer">Verify Node resources</a>.</p>
<p>Next step is to check logs, it's well described in <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/" rel="nofollow noreferrer">Troubleshoot Clusters</a> document. It's worth to mention that checking <code>kubelet logs</code> using <code>$ journalctl -u kubelet</code> also might provide very useful information.</p>
<p>More detailed node troubleshooting can be found also in documentations below:</p>
<ul>
<li><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/monitor-node-health/" rel="nofollow noreferrer">Apply Node problem detector</a></li>
</ul>
<blockquote>
<p>Node problem detector is a DaemonSet monitoring the node health. It collects node problems from various daemons and reports them to the apiserver as NodeCondition and Event.</p>
</blockquote>
<ul>
<li><a href="https://kubernetes.io/docs/setup/best-practices/node-conformance/" rel="nofollow noreferrer">Validate node setup</a></li>
<li><a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">Handling out of resources situation</a></li>
</ul>
