<p>My kubernetes K3s cluster gives this error:</p>
<pre><code>Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  17m   default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.
  Warning  FailedScheduling  17m   default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.
</code></pre>
<p>In order to list the taints in the cluster I executed:</p>
<pre><code>kubectl get nodes -o json | jq '.items[].spec'
</code></pre>
<p>which outputs:</p>
<pre><code>{
  &quot;podCIDR&quot;: &quot;10.42.0.0/24&quot;,
  &quot;podCIDRs&quot;: [
    &quot;10.42.0.0/24&quot;
  ],
  &quot;providerID&quot;: &quot;k3s://antonis-dell&quot;,
  &quot;taints&quot;: [
    {
      &quot;effect&quot;: &quot;NoSchedule&quot;,
      &quot;key&quot;: &quot;node.kubernetes.io/disk-pressure&quot;,
      &quot;timeAdded&quot;: &quot;2021-12-17T10:54:31Z&quot;
    }
  ]
}
{
  &quot;podCIDR&quot;: &quot;10.42.1.0/24&quot;,
  &quot;podCIDRs&quot;: [
    &quot;10.42.1.0/24&quot;
  ],
  &quot;providerID&quot;: &quot;k3s://knodea&quot;
}
</code></pre>
<p>When I use <code>kubectl describe node antonis-dell</code> I get:</p>
<pre><code>Name:               antonis-dell
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=antonis-dell
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        csi.volume.kubernetes.io/nodeid: {&quot;ch.ctrox.csi.s3-driver&quot;:&quot;antonis-dell&quot;}
                    flannel.alpha.coreos.com/backend-data: {&quot;VNI&quot;:1,&quot;VtepMAC&quot;:&quot;f2:d5:6c:6a:85:0a&quot;}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.1.XX
                    k3s.io/hostname: antonis-dell
                    k3s.io/internal-ip: 192.168.1.XX
                    k3s.io/node-args: [&quot;server&quot;]
                    k3s.io/node-config-hash: YANNMDBIL7QEFSZANHGVW3PXY743NWWRVFKBKZ4FXLV5DM4C74WQ====
                    k3s.io/node-env:
                      {&quot;K3S_DATA_DIR&quot;:&quot;/var/lib/rancher/k3s/data/e61cd97f31a54dbcd9893f8325b7133cfdfd0229ff3bfae5a4f845780a93e84c&quot;,&quot;K3S_KUBECONFIG_MODE&quot;:&quot;644&quot;}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 17 Dec 2021 12:11:39 +0200
Taints:             node.kubernetes.io/disk-pressure:NoSchedule
</code></pre>
<p>where it seems that node has a disk-pressure taint.</p>
<p>This command doesn't work: <code>kubectl taint node antonis-dell node.kubernetes.io/disk-pressure:NoSchedule-</code> and it seems to me that even if it worked, this is not a good solution because the taint assigned by the control plane.</p>
<p>Furthermore in the end of command <code>kubectl describe node antonis-dell</code> I observed this:</p>
<pre><code>Events:
  Type     Reason               Age                  From     Message
  ----     ------               ----                 ----     -------
  Warning  FreeDiskSpaceFailed  57m                  kubelet  failed to garbage collect required amount of images. Wanted to free 32967806976 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  52m                  kubelet  failed to garbage collect required amount of images. Wanted to free 32500092928 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  47m                  kubelet  failed to garbage collect required amount of images. Wanted to free 32190205952 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  42m                  kubelet  failed to garbage collect required amount of images. Wanted to free 32196628480 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  37m                  kubelet  failed to garbage collect required amount of images. Wanted to free 32190926848 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed  2m21s (x7 over 32m)  kubelet  (combined from similar events): failed to garbage collect required amount of images. Wanted to free 30909374464 bytes, but freed 0 bytes
</code></pre>
<p>Maybe the disk-pressure is related to this? How can I delete the unwanted images?</p>

e7lT2P
<p>Posting the answer as a community wiki, feel free to edit and expand.</p>
<hr />
<p><code>node.kubernetes.io/disk-pressure:NoSchedule</code> taint indicates that some disk pressure happens (as it's called).</p>
<blockquote>
<p>The <code>kubelet</code> detects disk pressure based on <code>imagefs.available</code>, <code>imagefs.inodesFree</code>, <code>nodefs.available</code> and <code>nodefs.inodesFree</code>(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the <code>kubelet</code> to determine if the Node condition and taint should be added/removed.</p>
</blockquote>
<p>More details on <code>disk-pressure</code> are available in <a href="https://medium.com/kubernetes-tutorials/efficient-node-out-of-resource-management-in-kubernetes-67f158da6e59" rel="nofollow noreferrer">Efficient Node Out-of-Resource Management in Kubernetes</a> under <code>How Does Kubelet Decide that Resources Are Low?</code> section:</p>
<blockquote>
<p><code>memory.available</code> — A signal that describes the state of cluster
memory. The default eviction threshold for the memory is 100 Mi. In
other words, the kubelet starts evicting Pods when the memory goes
down to 100 Mi.</p>
<p><code>nodefs.available</code> — The nodefs is a filesystem used by
the kubelet for volumes, daemon logs, etc. By default, the kubelet
starts reclaiming node resources if the nodefs.available &lt; 10%.</p>
<p><code>nodefs.inodesFree</code> — A signal that describes the state of the nodefs
inode memory. By default, the kubelet starts evicting workloads if the
nodefs.inodesFree &lt; 5%.</p>
<p><code>imagefs.available</code> — The imagefs filesystem is
an optional filesystem used by a container runtime to store container
images and container-writable layers. By default, the kubelet starts
evicting workloads if the imagefs.available &lt; 15 %.</p>
<p><code>imagefs.inodesFree</code> — The state of the imagefs inode memory. It has no default eviction threshold.</p>
</blockquote>
<hr />
<p><strong>What to check</strong></p>
<p>There are different things that can help, such as:</p>
<ul>
<li><p>prune unused objects like images (with Docker CRI) - <a href="https://docs.docker.com/config/pruning/#prune-images" rel="nofollow noreferrer">prune images</a>.</p>
<blockquote>
<p>The docker image prune command allows you to clean up unused images. By default, docker image prune only cleans up dangling images. A dangling image is one that is not tagged and is not referenced by any container.</p>
</blockquote>
</li>
<li><p>check files/logs on the node if they take a lot of space.</p>
</li>
<li><p>any another reason why disk space was consumed.</p>
</li>
</ul>
