<p>After migrating the image type from container-vm to cos for the nodes of a GKE cluster, it seems no longer possible to mount a NFS volume for a pod.</p>

<p>The problem seems to be missing NFS client libraries, as a mount command from command line fails on all COS versions I tried (cos-stable-58-9334-62-0, cos-beta-59-9460-20-0, cos-dev-60-9540-0-0).</p>

<pre><code>sudo mount -t nfs mynfsserver:/myshare /mnt
</code></pre>

<p>fails with</p>

<pre><code>mount: wrong fs type, bad option, bad superblock on mynfsserver:/myshare,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.&lt;type&gt; helper program)
</code></pre>

<p>But this contradicts the supported volume types listed here:
<a href="https://cloud.google.com/container-engine/docs/node-image-migration#storage_driver_support" rel="nofollow noreferrer">https://cloud.google.com/container-engine/docs/node-image-migration#storage_driver_support</a></p>

<p>Mounting a NFS volume in a pod works in a pool with image-type <code>container-vm</code> but not with <code>cos</code>.</p>

<p>With cos I get following messages with <code>kubectl describe pod</code>:</p>

<pre><code>MountVolume.SetUp failed for volume "kubernetes.io/nfs/b6e6cf44-41e7-11e7-8b00-42010a840079-nfs-mandant1" (spec.Name: "nfs-mandant1") pod "b6e6cf44-41e7-11e7-8b00-42010a840079" (UID: "b6e6cf44-41e7-11e7-8b00-42010a840079") with: mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1 nfs []
Output: Mount failed: Mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs singlefs-1-vm:/data/mandant1 /var/lib/kubelet/pods/b6e6cf44-41e7-11e7-8b00-42010a840079/volumes/kubernetes.io~nfs/nfs-mandant1]
Output: mount.nfs: Failed to resolve server singlefs-1-vm: Temporary failure in name resolution
</code></pre>

<p>I've nicked the solution @saad-ali mentioned above, from the kubernetes project, to make this work.</p>

<p>To be concrete, I've added the following to my cloud-config:</p>

<pre><code># This script creates a chroot environment containing the tools needed to mount an nfs drive
- path: /tmp/mount_config.sh
  permissions: 0755
  owner: root
  content: |
    #!/bin/sh
    set +x # For debugging

    export USER=root
    export HOME=/home/dockerrunner
    mkdir -p /tmp/mount_chroot
    chmod a+x /tmp/mount_chroot
    cd /tmp/
    echo "Sleeping for 30 seconds because toolbox pull fails otherwise"
    sleep 30
    toolbox --bind /tmp /google-cloud-sdk/bin/gsutil cp gs://&lt;uploaded-file-bucket&gt;/mounter.tar /tmp/mounter.tar
    tar xf /tmp/mounter.tar -C /tmp/mount_chroot/
    mount --bind /tmp/mount_chroot /tmp/mount_chroot
    mount -o remount, exec /tmp/mount_chroot
    mount --rbind /proc /tmp/mount_chroot/proc
    mount --rbind /dev /tmp/mount_chroot/dev
    mount --rbind /tmp /tmp/mount_chroot/tmp
    mount --rbind /mnt /tmp/mount_chroot/mnt
</code></pre>

<p>The uploaded-file-bucket container the chroom image the kube team has created, downloaded from: <a href="https://storage.googleapis.com/kubernetes-release/gci-mounter/mounter.tar" rel="nofollow noreferrer">https://storage.googleapis.com/kubernetes-release/gci-mounter/mounter.tar</a></p>

<p>Then, the runcmd for the cloud-config looks something like:</p>

<pre><code>runcmd:
- /tmp/mount_config.sh
- mkdir -p /mnt/disks/nfs_mount
- chroot /tmp/mount_chroot /bin/mount -t nfs -o rw nfsserver:/sftp /mnt/disks/nfs_mount
</code></pre>

<p>This works. Ugly as hell, but it'll have to do for now.</p>
