<p>We're running kubelet 1.18.3 with Docker 19.03.12 on a baremetal Linux cluster.  We noticed today that we had a lot of pod evictions that we traced to disk space pressure on a node.  For that particular node, we had passed the 80% threshold on the file system holding the Docker nodefs.</p>
<p>But the culprit was in /var/lib/kubelet/pods/{{pod-uid}}/volumes/kubernetes.io~nfs/{{pv}}.  On inspection, that directory held a copy. (a cache???) of an NFS persistent volume mounted ReadOnlyMany by the pod through a PersistentVolumeClaim. The pod is ultimately based on Debian Stretch and OpenJDK.  If we exec into the pod, we see the nfs mount point for the PV as we would expect:</p>
<pre><code>server:/export-path on /local-volume type nfs4 (ro,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=nn.nn.nn.nn,local_lock=none,addr=nn.nn.nn.nn)
</code></pre>
<p>However, <code>docker inspect -f '{{.Mounts}}' container-id</code> also shows the mount point <code>/local-volume</code> as being bound to /var/lib/kubelet/pods/{{pod-uid}}/volumes/kubernetes.io~nfs/{{pv}}</p>
<p>So two questions:</p>
<ol>
<li>AFAIK, nfsv4 client doesn't cache files in the mounted file system, so where are these copies coming from?</li>
<li>Is there a way to manage this local cache?  The external PV is pretty large, and we don't need it to be cached as a local volume in the pod.</li>
</ol>

Eric Schoen
<p>I should have looked deeper into the host's mtab.  Kubelet mounts the pod's persistent volume claim binding into <code>/var/lib/kubelet/pods/{{pod-uid}}/volumes/kubernetes.io~nfs/{{pv}}</code>, so the container's persistent volume appears in the host's file system.</p>
<p>Lesson learned.  Use <code>du -x -h -s * | sort -h</code> when hunting down big disk space offenders to not get distracted by (i.e., nfs) mount points.</p>
