<p>With kubernetes, I'm trying to deploy jenkins image &amp; a persistent volume mapped to a NFS share (which is mounted on all my workers)</p>

<ul>
<li>So, this is my share on my workers :</li>
</ul>

<pre><code>[root@pp-tmp-test24 /opt]# df -Th /opt/jenkins.persistent
Filesystem                                        Type  Size  Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP nfs4   10G  9.5M   10G   1% /opt/jenkins.persistent
</code></pre>

<ul>
<li>And My data on this share</li>
</ul>

<pre><code>[root@pp-tmp-test24 /opt/jenkins.persistent]# ls -l
total 0
-rwxr-xr-x. 1 root root 0 Oct  2 11:53 newfile

[root@pp-tmp-test24 /opt/jenkins.persistent]# cat newfile
hello
</code></pre>

<ul>
<li>Here It is my yaml files to deploy it</li>
</ul>

<p>My PersistentVolume yaml</p>

<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv-nfs
  labels:
    type: type-nfs
spec:
  storageClassName: class-nfs
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  hostPath:
    path: /opt/jenkins.persistent
</code></pre>

<p>My PersistentVolumeClaim yaml</p>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc-nfs
  namespace: ns-jenkins
spec:
  storageClassName: class-nfs
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      type: type-nfs
</code></pre>

<p>And my deployment</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: ns-jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - image: jenkins
     #- image: httpd:latest
        name: jenkins
        ports:
        - containerPort: 8080
          protocol: TCP
          name: jenkins-web
        volumeMounts:
        - name: jenkins-persistent-storage
          mountPath: /var/foo
      volumes:
      - name: jenkins-persistent-storage
        persistentVolumeClaim:
          claimName: jenkins-pvc-nfs
</code></pre>

<ul>
<li>After <code>kubectl create -f</code> command, all is looking good :</li>
</ul>

<pre><code># kubectl get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
jenkins-pv-nfs   10Gi       RWX            Recycle          Bound    ns-jenkins/jenkins-pvc-nfs   class-nfs               37s
</code></pre>

<pre><code># kubectl get pvc -A
NAMESPACE    NAME              STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ns-jenkins   jenkins-pvc-nfs   Bound    jenkins-pv-nfs   10Gi       RWX            class-nfs      35s
</code></pre>

<pre><code># kubectl get pods -A |grep jenkins
ns-jenkins             jenkins-5bdb8678c-x6vht                                                  1/1     Running   0          14s
</code></pre>

<pre><code># kubectl describe pod jenkins-5bdb8678c-x6vht -n ns-jenkins

Name:           jenkins-5bdb8678c-x6vht
Namespace:      ns-jenkins
Priority:       0
Node:           pp-tmp-test25.mydomain/172.31.68.225
Start Time:     Wed, 02 Oct 2019 11:48:23 +0200
Labels:         app=jenkins
                pod-template-hash=5bdb8678c
Annotations:    &lt;none&gt;
Status:         Running
IP:             10.244.5.47
Controlled By:  ReplicaSet/jenkins-5bdb8678c
Containers:
  jenkins:
    Container ID:   docker://8a3e4871ed64b371818bac59e24d6912e5d2b13c8962c1639d36797fbce8082e
    Image:          jenkins
    Image ID:       docker-pullable://docker.io/jenkins@sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 02 Oct 2019 11:48:26 +0200
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /var/foo from jenkins-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dz6cd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  jenkins-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins-pvc-nfs
    ReadOnly:   false
  default-token-dz6cd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dz6cd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                     Message
  ----    ------     ----  ----                                                     -------
  Normal  Scheduled  39s   default-scheduler                                        Successfully assigned ns-jenkins/jenkins-5bdb8678c-x6vht to pp-tmp-test25.mydomain
  Normal  Pulling    38s   kubelet, pp-tmp-test25.mydomain  Pulling image "jenkins"
  Normal  Pulled     36s   kubelet, pp-tmp-test25.mydomain  Successfully pulled image "jenkins"
  Normal  Created    36s   kubelet, pp-tmp-test25.mydomain  Created container jenkins
  Normal  Started    36s   kubelet, pp-tmp-test25.mydomain  Started container jenkins
</code></pre>

<ul>
<li>On my worker, this is my container </li>
</ul>

<pre><code># docker ps |grep jenkins
8a3e4871ed64        docker.io/jenkins@sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668           "/bin/tini -- /usr..."   2 minutes ago       Up 2 minutes                            k8s_jenkins_jenkins-5bdb8678c-x6vht_ns-jenkins_64b66dae-a1da-4d90-83fd-ff433638dc9c_0
</code></pre>

<p>So I launch a shell on my container, and I can see my data on <code>/var/foo</code> :</p>

<pre><code># docker exec -t -i 8a3e4871ed64 /bin/bash


jenkins@jenkins-5bdb8678c-x6vht:/$ df -h /var/foo
Filesystem                                                                                           Size  Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP                                                     10G  9.5M   10G   1% /var/foo


jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ ls -lZ /var/foo -d
drwxr-xr-x. 2 root root system_u:object_r:nfs_t:s0 4096 Oct  2 10:06 /var/foo


jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ ls -lZ /var/foo
-rwxr-xr-x. 1 root root system_u:object_r:nfs_t:s0 12 Oct  2 10:05 newfile

jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ cat newfile
hello
</code></pre>

<p>I'm trying to write data in my <code>/var/foo/newfile</code> but the Permission is denied</p>

<pre><code>jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ echo "world" &gt;&gt; newfile
bash: newfile: Permission denied
</code></pre>

<p>Same thing in  my <code>/var/foo/ directory</code>, I can't write data </p>

<pre><code>jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ touch newfile2
touch: cannot touch 'newfile2': Permission denied
</code></pre>

<p>So, I tried an another image like <code>httpd:latest</code> in my deployment yaml (keeping the same name in my yaml definition)</p>

<pre><code>[...]
      containers:
      #- image: jenkins
      - image: httpd:latest
[...]
</code></pre>

<pre><code># docker ps |grep jenkins
fa562400405d        docker.io/httpd@sha256:39d7d9a3ab93c0ad68ee7ea237722ed1b0016ff6974d80581022a53ec1e58797             "httpd-foreground"       50 seconds ago      Up 48 seconds                           k8s_jenkins_jenkins-7894877f96-6dj85_ns-jenkins_540b12bd-69df-44d8-b3df-20a0a96cc851_0
</code></pre>

<p>In my new container, this time I can Read-Write data :</p>

<pre><code>root@jenkins-7894877f96-6dj85:/usr/local/apache2# df -h /var/foo
Filesystem                                         Size  Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP   10G  9.6M   10G   1% /var/foo

root@jenkins-7894877f96-6dj85:/var/foo# ls -lZ
total 0
-rwxr-xr-x. 1 root root system_u:object_r:nfs_t:s0 12 Oct  2 10:05 newfile
-rw-r--r--. 1 root root system_u:object_r:nfs_t:s0  0 Oct  2 10:06 newfile2

root@jenkins-7894877f96-6dj85:/var/foo# ls -lZ /var/foo -d
drwxr-xr-x. 2 root root system_u:object_r:nfs_t:s0 4096 Oct  2 10:06 /var/foo


root@jenkins-7894877f96-6dj85:/var/foo# ls -l
total 0
-rwxr-xr-x. 1 root root 6 Oct  2 09:55 newfile

root@jenkins-7894877f96-6dj85:/var/foo# echo "world" &gt;&gt; newfile
root@jenkins-7894877f96-6dj85:/var/foo# touch newfile2
root@jenkins-7894877f96-6dj85:/var/foo# ls -l
total 0
-rwxr-xr-x. 1 root root 12 Oct  2 10:05 newfile
-rw-r--r--. 1 root root  0 Oct  2 10:06 newfile2
</code></pre>

<p>What I'm doing wrong ? Does the pb is due to <code>jenkins</code> images who do not allow RW access ? Same pb with a local storage (on my worker) with persistent volume.</p>

<p>Other thing, perhaps it is stupid :  with my jenkins image, I would like to mount the <code>/var/jenkins_home</code> dir to a persistent volume in order to keep jenkins's configuration files. But if I try to mount <code>/var/jenkins_home</code> instead of <code>/var/foo</code>, pod is crashinglookbackoff (because there is already data stored in <code>/var/jenkins_home</code>).</p>

<p>thank you all for your help !</p>

Vincent
<p>I noticed You are trying to write as <code>jenkins</code> user on <code>jenkins-5bdb8678c-x6vht</code> that might not have write permissions in that root:root directory.</p>

<p>You might want to change that directory permissions to match <code>jenkins</code> user privileges.</p>

<p>Try to verify that this is causing this issue by using <code>sudo</code> before writing to file.</p>

<p>If you <code>sudo</code> is not installed then exec in with <code>--user</code> flag as <code>root</code> user. So its just like in other cases where writing worked.</p>

<p><code>docker exec -t -i -u root 8a3e4871ed64 /bin/bash</code></p>
