<p>I've set up an nfs server that serves a RMW pv according to the example at <a href="https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs" rel="nofollow noreferrer">https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs</a></p>
<p>This setup works fine for me in lots of production environments, but in some specific GKE cluster instance, mount stopped working after pods restarted.</p>
<p>From kubelet logs I see the following repeating many times</p>
<blockquote>
<p>Unable to attach or mount volumes for pod &quot;api-bf5869665-zpj4c_default(521b43c8-319f-425f-aaa7-e05c08282e8e)&quot;: unmounted volumes=[shared-mount], unattached volumes=[geekadm-net deployment-role-token-6tg9p shared-mount]: timed out waiting for the condition; skipping pod</p>
</blockquote>
<blockquote>
<p>Error syncing pod 521b43c8-319f-425f-aaa7-e05c08282e8e (&quot;api-bf5869665-zpj4c_default(521b43c8-319f-425f-aaa7-e05c08282e8e)&quot;), skipping: unmounted volumes=[shared-mount], unattached volumes=[geekadm-net deployment-role-token-6tg9p shared-mount]: timed out waiting for the condition</p>
</blockquote>
<p>Manually mounting the nfs on any of the nodes work just fine: <code>mount -t nfs &lt;service ip&gt;:/ /tmp/mnt</code></p>
<p>How can I further debug the issue? Are there any other logs I could look at besides kubelet?</p>

<p>In case the pod gets kicked out of the node because the mount is too slow, you may see messages like that in logs.</p>
<p>Kubelets even inform about this issue in logs.<br />
<strong>Sample log from Kubelets:</strong><br />
Setting volume ownership for /var/lib/kubelet/pods/c9987636-acbe-4653-8b8d-
aa80fe423597/volumes/kubernetes.io~gce-pd/pvc-fbae0402-b8c7-4bc8-b375-
1060487d730d and fsGroup set. If the volume has a lot of files then setting
volume ownership could be slow, see <a href="https://github.com/kubernetes/kubernetes/issues/69699" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/69699</a></p>
<p><strong>Cause:</strong><br />
The pod.spec.securityContext.fsGroup setting causes kubelet to run chown and chmod on all the files in the volumes mounted for given pod. This can be a very time consuming thing to do in case of big volumes with many files.</p>
<p>By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. From the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods" rel="nofollow noreferrer">document</a>.</p>
<p><strong>Solution:</strong><br />
You can deal with it in the following ways.</p>
<ol>
<li>Reduce the number of files in the volume.</li>
<li>Stop using the fsGroup setting.</li>
</ol>
