<p dir="auto">I premise I'm opening a ticket only after almost 48 hours of searches &amp; tests.<br>
I tested even your official NFS examples and unofficial docker images and configs - nothing works.</p>
<p dir="auto"><strong>Kubernetes version</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.7&quot;, GitCommit:&quot;a2cba278cba1f6881bb0a7704d9cac6fca6ed435&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-12T23:15:30Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0&quot;, GitCommit:&quot;a16c0a7f71a6f93c7e0f222d961f4675cd97a46b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T18:10:32Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li>Google Cloud - Container Engine Cluster (europe-west1-b   , 1 node, 1 vCPU, RAM 1.7 GB, node ver. 1.4.0 ) - All permissions on (full/enabled/read-write)</li>
<li>Every command &amp; test is running through Google Cloud Shell VM.</li>
</ul>
<p dir="auto"><strong>The Problem:</strong><br>
I'm trying to run the same environment (which was working on previous versions of k8s) on Google Container Engine with this config:<br>
<strong>1 deployment (1 replica container) as NFS server</strong> connected directly to Google Persistent Disk (Standard - 10Gb) - gcr.io/google_containers/volume-nfs image<br>
<strong>1 deployment (1 or 2 replicas / containers) as web server</strong> - php:7.0.8-apache official docker image</p>
<p dir="auto"><strong>The main scope is to share 1 pd-volume between each webapp-server replica</strong> but when I create my deployments, during the web-app deployment creation, it breaks with NFS/mount problem.<br>
(the NFS server starts even if with some warning)</p>
<p dir="auto"><strong>Here some other info about the mount error:</strong></p>
<p dir="auto"><code class="notranslate">kubectl describe po web-app</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="....[omitted]...
Mounting arguments: 10.0.2.9:/exports /var/lib/kubelet/pods/...../volumes/kubernetes.io~nfs/pv-nfs-webapp-logs nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.0.2.9:/exports,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.&lt;type&gt; helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

....[omitted]..."><pre class="notranslate"><code class="notranslate">....[omitted]...
Mounting arguments: 10.0.2.9:/exports /var/lib/kubelet/pods/...../volumes/kubernetes.io~nfs/pv-nfs-webapp-logs nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.0.2.9:/exports,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.&lt;type&gt; helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

....[omitted]...
</code></pre></div>
<p dir="auto">From this moment <code class="notranslate">10.0.2.9</code> is the NFS Server host IP</p>
<p dir="auto">If I connect inside the web-app pod I can ping the NFS server (by IP), and it's sharing the right folders as described:<br>
<code class="notranslate">showmount -e 10.0.2.9</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Export list for 10.0.2.9:
/        *
/exports *"><pre class="notranslate"><code class="notranslate">Export list for 10.0.2.9:
/        *
/exports *
</code></pre></div>
<p dir="auto">The nmap report confirms running ports are OK:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Nmap scan report for 10.0.2.9
Host is up (0.000026s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE
111/tcp  open  rpcbind
2049/tcp open  nfs"><pre class="notranslate"><code class="notranslate">Nmap scan report for 10.0.2.9
Host is up (0.000026s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE
111/tcp  open  rpcbind
2049/tcp open  nfs
</code></pre></div>
<p dir="auto"><strong>Here the logs of NFS pod:</strong><br>
<code class="notranslate">kubectl logs nfs-deployment</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Serving /exports
Serving /
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: / does not support NFS export
NFS started"><pre class="notranslate"><code class="notranslate">Serving /exports
Serving /
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: / does not support NFS export
NFS started
</code></pre></div>
<p dir="auto"><code class="notranslate">kubectl describe po nfs-deployment</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Name:           nfs-deployment-...
Status:         Running
IP:             10.0.2.9
...
Controllers:    ReplicaSet/nfs-deployment-115940229
Containers:
  nfs-server:
    Container ID:       docker://.....
    Image:              gcr.io/google_containers/volume-nfs
    Image ID:           docker://sha256:...
    Ports:              2049/TCP, 111/TCP
    Requests:
      cpu:                      10m
    State:                      Running
      Started:                  Thu, 06 Oct 2016 16:45:40 +0200
    Ready:                      True
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True 
  Ready         True 
  PodScheduled  True 
Volumes:
  nfs-pd:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     disk-1
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-f7r08:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-f..."><pre class="notranslate"><code class="notranslate">Name:           nfs-deployment-...
Status:         Running
IP:             10.0.2.9
...
Controllers:    ReplicaSet/nfs-deployment-115940229
Containers:
  nfs-server:
    Container ID:       docker://.....
    Image:              gcr.io/google_containers/volume-nfs
    Image ID:           docker://sha256:...
    Ports:              2049/TCP, 111/TCP
    Requests:
      cpu:                      10m
    State:                      Running
      Started:                  Thu, 06 Oct 2016 16:45:40 +0200
    Ready:                      True
    Restart Count:              0
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True 
  Ready         True 
  PodScheduled  True 
Volumes:
  nfs-pd:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     disk-1
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-f7r08:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-f...
</code></pre></div>
<p dir="auto">I tested many configurations but the same working on older k8s version is like this:<br>
(in order of creation)</p>
<p dir="auto"><strong>nfs.yaml</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# NFS POD / SERVER (PERSISTEN DISK SHARING)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs-server
        role: backend
        #env: staging
    spec:
      restartPolicy: Always
      containers:
      - name: nfs-server
        #image: gcr.io/google-samples/nfs-server:1.1
        image: gcr.io/google_containers/volume-nfs
        #imagePullPolicy: Always
        ports:
          #- name: nfs
            # containerPort: 2049
            # hostPort: 2049
          - name: nfs
            hostPort: 2049
            containerPort: 2049
            protocol: TCP
          - name: mountd
            hostPort: 20048
            containerPort: 20048
            protocol: UDP
          - name: rpcbind
            hostPort: 111
            containerPort: 111
        volumeMounts:
          - mountPath: &quot;/exports&quot;
            name: nfs-pd
        securityContext:
            privileged: true
        resources:
          requests:
            cpu: 10m
      volumes:
        - name: nfs-pd
          gcePersistentDisk:
            pdName: disk-1
            fsType: ext4"><pre class="notranslate"><code class="notranslate"># NFS POD / SERVER (PERSISTEN DISK SHARING)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nfs-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs-server
        role: backend
        #env: staging
    spec:
      restartPolicy: Always
      containers:
      - name: nfs-server
        #image: gcr.io/google-samples/nfs-server:1.1
        image: gcr.io/google_containers/volume-nfs
        #imagePullPolicy: Always
        ports:
          #- name: nfs
            # containerPort: 2049
            # hostPort: 2049
          - name: nfs
            hostPort: 2049
            containerPort: 2049
            protocol: TCP
          - name: mountd
            hostPort: 20048
            containerPort: 20048
            protocol: UDP
          - name: rpcbind
            hostPort: 111
            containerPort: 111
        volumeMounts:
          - mountPath: "/exports"
            name: nfs-pd
        securityContext:
            privileged: true
        resources:
          requests:
            cpu: 10m
      volumes:
        - name: nfs-pd
          gcePersistentDisk:
            pdName: disk-1
            fsType: ext4
</code></pre></div>
<p dir="auto"><strong>pvc.yaml</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-webapp
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    #this will be changed after nfs server is ready
    server: 10.0.2.7
    path: &quot;/exports&quot;
    readOnly: false

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nfs-webapp
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi"><pre class="notranslate"><code class="notranslate">apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-webapp
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    #this will be changed after nfs server is ready
    server: 10.0.2.7
    path: "/exports"
    readOnly: false

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nfs-webapp
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
</code></pre></div>
<p dir="auto"><strong>web.yaml</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# POD DEPLOYMENT
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: webapp-deployment
spec:
  #replicas: 1
  template:
    metadata:
      labels:
        app: webapp
        role: frontend
    spec:
      restartPolicy: Always
      containers:
      - name: webapp
        image: gcr.io/my-app-reg/my-webapp:v0.0.1
        ports:
          - name: apache-http
            containerPort: 80
          - name: apache-https
            containerPort: 443
        resources:
          requests:
            cpu: 10m
        livenessProbe:
          httpGet:
            path: /kubernetes-api/status/healthz
            port: 80
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /kubernetes-api/status/readiness
            port: 80
            scheme: HTTP
          initialDelaySeconds: 15
          timeoutSeconds: 10
        volumeMounts:
          - name: nfs-vol-webapp
          mountPath: /var/www/html/
      volumes:
        - name: nfs-vol-webapp
          persistentVolumeClaim:
            claimName: pvc-nfs-webapp"><pre class="notranslate"><code class="notranslate"># POD DEPLOYMENT
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: webapp-deployment
spec:
  #replicas: 1
  template:
    metadata:
      labels:
        app: webapp
        role: frontend
    spec:
      restartPolicy: Always
      containers:
      - name: webapp
        image: gcr.io/my-app-reg/my-webapp:v0.0.1
        ports:
          - name: apache-http
            containerPort: 80
          - name: apache-https
            containerPort: 443
        resources:
          requests:
            cpu: 10m
        livenessProbe:
          httpGet:
            path: /kubernetes-api/status/healthz
            port: 80
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        readinessProbe:
          httpGet:
            path: /kubernetes-api/status/readiness
            port: 80
            scheme: HTTP
          initialDelaySeconds: 15
          timeoutSeconds: 10
        volumeMounts:
          - name: nfs-vol-webapp
          mountPath: /var/www/html/
      volumes:
        - name: nfs-vol-webapp
          persistentVolumeClaim:
            claimName: pvc-nfs-webapp
</code></pre></div>
<p dir="auto">Relevant test failures: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="178984773" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33401" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/33401/hovercard" href="https://github.com/kubernetes/kubernetes/issues/33401">#33401</a></p>
<p dir="auto">I ran the e2e that "create a PV and a pre-bound PVC: test write access" and it fails on GCI.<br>
I debugged a bit and figured that the mount command is failing -</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Sep 24 21:17:31.083: INFO: At 2016-09-24 21:11:05 -0700 PDT - event for write-pod-1mzyy: {kubelet e2e-test-vishnuk-minion-group-jo
4u} FailedMount: MountVolume.SetUp failed for volume &quot;kubernetes.io/nfs/13fd01f0-82d6-11e6-959f-42010af00002-nfs-1nmbf&quot; (spec.Name
: &quot;nfs-1nmbf&quot;) pod &quot;13fd01f0-82d6-11e6-959f-42010af00002&quot; (UID: &quot;13fd01f0-82d6-11e6-959f-42010af00002&quot;) with: mount failed: exit s
tatus 32
Mounting arguments: 10.180.0.10:/exports /var/lib/kubelet/pods/13fd01f0-82d6-11e6-959f-42010af00002/volumes/kubernetes.io~nfs/nfs-
1nmbf nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.180.0.10:/exports,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.&lt;type&gt; helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so."><pre class="notranslate">Sep 24 21:17:31.083: INFO: At 2016-09-24 21:11:05 -0700 PDT - event <span class="pl-k">for</span> write-pod-1mzyy: {kubelet e2e-test-vishnuk-minion-group-jo
4u} FailedMount: MountVolume.SetUp failed <span class="pl-k">for</span> volume <span class="pl-s"><span class="pl-pds">"</span>kubernetes.io/nfs/13fd01f0-82d6-11e6-959f-42010af00002-nfs-1nmbf<span class="pl-pds">"</span></span> (spec.Name
<span class="pl-c1">:</span> <span class="pl-s"><span class="pl-pds">"</span>nfs-1nmbf<span class="pl-pds">"</span></span>) pod <span class="pl-s"><span class="pl-pds">"</span>13fd01f0-82d6-11e6-959f-42010af00002<span class="pl-pds">"</span></span> (UID: <span class="pl-s"><span class="pl-pds">"</span>13fd01f0-82d6-11e6-959f-42010af00002<span class="pl-pds">"</span></span>) with: mount failed: <span class="pl-c1">exit</span> s
tatus 32
Mounting arguments: 10.180.0.10:/exports /var/lib/kubelet/pods/13fd01f0-82d6-11e6-959f-42010af00002/volumes/kubernetes.io~nfs/nfs-
1nmbf nfs []
Output: mount: wrong fs type, bad option, bad superblock on 10.180.0.10:/exports,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<span class="pl-k">&lt;</span>type<span class="pl-k">&gt;</span> helper program)

       In some cases useful info is found <span class="pl-k">in</span> syslog - try
       dmesg <span class="pl-k">|</span> tail or so.</pre></div>
<p dir="auto">Looking at the kernel logs, I got the following information:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered named UNIX socket transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered udp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize umh client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize legacy client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: Unable to initialize client recovery tracking! (-22)
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: starting 10-second grace period (net ffff8801e8941840)"><pre lang="text" class="notranslate"><code class="notranslate">Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered named UNIX socket transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered udp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize umh client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: attempt to initialize legacy client tracking in a container ignored.
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: Unable to initialize client recovery tracking! (-22)
Sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: NFSD: starting 10-second grace period (net ffff8801e8941840)
</code></pre></div>
<p dir="auto">I looked at the NFS server logs and found the following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Serving /exports
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: /exports does not support NFS export
NFS started"><pre class="notranslate"><code class="notranslate">Serving /exports
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused
Starting rpcbind
exportfs: /exports does not support NFS export
NFS started
</code></pre></div>
<p dir="auto">I'd like to know if this is a test setup issue or a base image issue? NFS has been confirmed to be working on GCI by other users of GCI. Even then, I'd like to understand the reason for the NFS PV test failure in GCI.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/matchstick/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/matchstick">@matchstick</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/saad-ali/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/saad-ali">@saad-ali</a> @kubernetes/sig-storage <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/thockin/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/thockin">@thockin</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Amey-D/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Amey-D">@Amey-D</a></p>