<p>I have a node in my K8S cluster that I use for monitoring tools.</p>

<p>Pods running here: <code>Grafana</code>, <code>PGAdmin</code>, <code>Prometheus</code>, and <code>kube-state-metrics</code> </p>

<p>My problem is that I have a lot of evicted pods</p>

<p>The pods evicted: <code>kube-state-metrics</code>, <code>grafana-core</code>, <code>pgadmin</code></p>

<p>Then, the pod evicted with reason: <code>The node was low on resource: [DiskPressure].</code> : <code>kube-state-metrics</code> (90% of evicted pods), <code>pgadmin</code> (20% of evicted pods)</p>

<p>When I check any of the pods, I have free space on disk:</p>

<pre><code>bash-5.0$ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                   7.4G      3.3G      3.7G  47% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                   484.2M         0    484.2M   0% /sys/fs/cgroup
/dev/nvme0n1p2            7.4G      3.3G      3.7G  47% /dev/termination-log
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/nvme0n1p2            7.4G      3.3G      3.7G  47% /etc/resolv.conf
/dev/nvme0n1p2            7.4G      3.3G      3.7G  47% /etc/hostname
/dev/nvme0n1p2            7.4G      3.3G      3.7G  47% /etc/hosts
/dev/nvme2n1            975.9M      8.8M    951.1M   1% /var/lib/grafana
/dev/nvme0n1p2            7.4G      3.3G      3.7G  47% /etc/grafana/provisioning/datasources
tmpfs                   484.2M     12.0K    484.2M   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   484.2M         0    484.2M   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                   484.2M         0    484.2M   0% /sys/firmware
</code></pre>

<p>Only one or two pods show another message:</p>

<pre><code>The node was low on resource: ephemeral-storage. Container addon-resizer was using 48Ki, which exceeds its request of 0. Container kube-state-metrics was using 44Ki, which exceeds its request of 0.

The node was low on resource: ephemeral-storage. Container pgadmin was using 3432Ki, which exceeds its request of 0.
</code></pre>

<p>I also have kubelet saying: </p>

<pre><code>(combined from similar events): failed to garbage collect required amount of images. Wanted to free 753073356 bytes, but freed 0 bytes
</code></pre>

<p>I have those pods running on a AWS <code>t3.micro</code></p>

<p>It appears that it is not affecting my services in production. </p>

<p>Why is it happening, and how should I fix this.</p>

<p>EDIT: Here is the result when I do <code>df -h</code> in my node</p>

<pre><code>admin@ip-172-20-41-112:~$ df -h 
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           789M  3.0M  786M   1% /run
/dev/nvme0n1p2  7.5G  6.3G  804M  89% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
</code></pre>

<p>I can see that <code>/dev/nvme0n1p2</code>, but how can I see the content ? when I do <a href="https://dev.yorhel.nl/ncdu" rel="nofollow noreferrer">ncdu</a> in /, I can only see 3GB of data... </p>

<p>Apparently you're about to run out of the available disk space <strong>on your node</strong>. However keep in mind that according to the <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-conditions" rel="noreferrer">documentation</a> <code>DiskPressure</code> condition denotes:</p>

<blockquote>
  <p>Available disk space and inodes on either the node’s root filesystem
  or image filesystem has satisfied an eviction threshold</p>
</blockquote>

<p>Try to run <code>df -h</code> but on your worker <code>node</code>, not in a <code>Pod</code>. What is the percentage of disk usage ? Additionally you may check <strong>kubelet</strong> logs for more details:</p>

<pre><code>journalctl -xeu kubelet.service
</code></pre>

<p>Also take a look at <a href="https://medium.com/faun/kubelet-pod-the-node-was-low-on-resource-diskpressure-384f590892f5" rel="noreferrer">this</a> article and <a href="https://github.com/kubernetes/kubernetes/issues/66961#issuecomment-448135500" rel="noreferrer">this</a> comment.</p>

<p>Let me know if it helps.</p>

<p><a href="https://stackoverflow.com/questions/42576661/diskpressure-crashing-the-node">Here</a> you can find an answer which explains very well the same topic.</p>

<h3>update:</h3>

<p>This line clearly shows that the default treshold is close to being exceeded:</p>

<pre><code>/dev/nvme0n1p2  7.5G  6.3G  804M  89% /
</code></pre>

<p>Swith to the root user ( <code>su -</code> ) and run:</p>

<pre><code>du -hd1 /
</code></pre>

<p>to see what directories take up most of the disk space.</p>
