<p>We noticed that these errors started after the node-pool was autoscaled and the existing Nodes were replaced with new compute instances. This also happened during a maintenance window. We're using an NFS server. GCP Kubernetes Cluster version is 1.21.6</p>
<p>The issue appears to have only affect certain Nodes on the cluster. We've cordoned the Nodes where the mount error's and pods on the &quot;healthy&quot; nodes are working.</p>
<pre><code>&quot;Unable to attach or mount volumes: unmounted volumes=[vol],
unattached volumes=[vol]: timed out waiting for the condition&quot;
</code></pre>
<p>We're also seeing errors on the konnectivity-agent:</p>
<pre><code>&quot;connection read failure&quot; err=&quot;read tcp
10.4.2.34:43682-&gt;10.162.0.119:10250: use of closed network connection&quot;
</code></pre>
<p>We believe the issue is when autoscaling is enabled, and new Nodes are introduced to the pool. The problem is it appears to be completely random. Sometimes the pods come up fine and others get the mount error.</p>

westcoastdev
<p>This Error  indicates that the NFS workload would be stuck at a Terminating state. Moreover, some disk throttling might be observed on worker nodes.</p>
<p><strong>Solution:</strong></p>
<p>There are two possible workarounds for this issue.</p>
<ul>
<li><p><a href="https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#force-deletion" rel="nofollow noreferrer">Force Deletion</a> of the NFS workload can sometimes mitigate the issue.
After deletion, you may also need to restart the Kubelet of the
worker node.</p>
</li>
<li><p>NFS versions v4.1 and v4.2 shouldn't be affected by this issue. The
NFS version is specified via <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes" rel="nofollow noreferrer">configuration</a> and doesn't require an
image change.</p>
</li>
</ul>
<p>Please change the NFS version as seen below:</p>
<pre><code>mountOptions:



 - nfsvers=4.2
</code></pre>
<p><strong>Cause</strong></p>
<p>NFS v4.0 has a known issue which is a limitation of the NFS pod when there are too many connections at once. Because of that limitation, the Kubelet can't unmount the NFS volume from the pod and the worker node if the NFS container was deleted first.
Moreover, NFS mounts going stale when the server dies is a <a href="https://github.com/kubernetes/kubernetes/issues/72048" rel="nofollow noreferrer">known issue</a>. It is known that many of these stale mounts building upon a worker node can cause future NFS mounts to slow down. Please note that there can also be some unmount issue related to this error.</p>
