<p>The spring boot application is deployed on openshift 4. This application needs to create a file on the nfs-share.
The openshift container has configured a volume mount on the type NFS.
The container on openshift creates a pod with random userid as</p>
<pre><code>sh-4.2$ id
uid=1031290500(1031290500) gid=0(root) groups=0(root),1031290500
</code></pre>
<p>The mount point is /nfs/abc</p>
<pre><code>sh-4.2$ ls -la /nfs/
ls: cannot access /nfs/abc: Permission denied
total 0
drwxr-xr-x. 1 root root 29 Nov 25 09:34 .
drwxr-xr-x. 1 root root 50 Nov 25 10:09 ..
d?????????? ? ?    ?     ?            ? abc
</code></pre>
<p>on the docker image I created a user &quot;technical&quot; with uid= gid=48760 as shown below.</p>
<pre><code>FROM quay.repository
MAINTAINER developer

LABEL description=&quot;abc image&quot; \
      name=&quot;abc&quot; \
      version=&quot;1.0&quot;

ARG APP_HOME=/opt/app
ARG PORT=8080

ENV JAR=app.jar \
    SPRING_PROFILES_ACTIVE=default \
    JAVA_OPTS=&quot;&quot;

    
RUN mkdir $APP_HOME 

ADD $JAR $APP_HOME/

WORKDIR $APP_HOME
EXPOSE $PORT
ENTRYPOINT java $JAVA_OPTS -Dspring.profiles.active=$SPRING_PROFILES_ACTIVE -jar $JAR
</code></pre>
<p>my deployment config file is as shown below</p>
<pre><code> spec:
      volumes:
        - name: bad-import-file
          persistentVolumeClaim:
            claimName: nfs-test-pvc
      containers:
        - resources:
            limits:
              cpu: '1'
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          name: abc
          env:
            - name: SPRING_PROFILES_ACTIVE
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: spring.profiles.active
            - name: DB_URL
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: db.url
            - name: DB_USERNAME
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: db.username
            - name: BAD_IMPORT_PATH
              valueFrom:
                configMapKeyRef:
                  name: abc-configmap
                  key: bad.import.path
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: abc-secret
                  key: db.password
          ports:
            - containerPort: 8080
              protocol: TCP
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: bad-import-file
              mountPath: /nfs/abc
      dnsPolicy: ClusterFirst
      securityContext:
        runAsGroup: 44337
        runAsNonRoot: true
        supplementalGroups:
          - 44337
</code></pre>
<p>the PV request is as follows</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: abc-tuc-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: classic-nfs
  mountOptions:
    - hard
    - nfsvers=3
  nfs:
    path: /tm03v06_vol3014
    server: tm03v06cl02.jit.abc.com
    readOnly: false
</code></pre>
<p>Now the openshift user has id</p>
<pre><code>sh-4.2$ id
        uid=1031290500(1031290500) gid=44337(technical) groups=44337(technical),1031290500
</code></pre>
<p><strong>RECENT UPDATE</strong></p>
<p>Just to be clear with the problem, Below I have two commands from the same pod terminal,</p>
<pre><code>sh-4.2$ cd /nfs/
sh-4.2$ ls -la (The first command I tried immediately after pod creation.)
total 8
drwxr-xr-x.  1 root  root    29 Nov 29 08:20 .
drwxr-xr-x.  1 root  root    50 Nov 30 08:19 ..
drwxrwx---. 14 technical technical 8192 Nov 28 19:06 abc
sh-4.2$ ls -la(few seconds later on the same pod terminal)
ls: cannot access abc: Permission denied
total 0
drwxr-xr-x. 1 root root 29 Nov 29 08:20 .
drwxr-xr-x. 1 root root 50 Nov 30 08:19 ..
d?????????? ? ?    ?     ?            ? abc
</code></pre>
<p>So the problem is that I see these question marks(???) on the mount point.
The mounting is working correctly but I cannot access this /nfs/abc directory and I see this ????? for some reason</p>
<p>UPDATE</p>
<pre><code>sh-4.2$ ls -la /nfs/abc/
ls: cannot open directory /nfs/abc/: Stale file handle
sh-4.2$ ls -la /nfs/abc/ (after few seconds on the same pod terminal)
ls: cannot access /nfs/abc/: Permission denied
</code></pre>
<p>Could this STALE FILE HANDLE be the reason for this issue?</p>

<h1>TL;DR</h1>
<p>You can use the <code>anyuid</code> security context to run the pod to avoid having OpenShift assign an arbitrary UID, and set the permissions on the volume to the known UID of the user.</p>
<hr />
<p><a href="https://cookbook.openshift.org/users-and-role-based-access-control/why-do-my-applications-run-as-a-random-user-id.html" rel="nofollow noreferrer">OpenShift will override the user ID the image itself may specify that it should run as</a>:</p>
<blockquote>
<p>The user ID isn't actually entirely random, but is an assigned user ID which is unique to your project. In fact, your project is assigned a range of user IDs that applications can be run as. The set of user IDs will not overlap with other projects. You can see what range is assigned to a project by running oc describe on the project.</p>
</blockquote>
<blockquote>
<p>The purpose of assigning each project a distinct range of user IDs is so that in a multitenant environment, applications from different projects never run as the same user ID. When using persistent storage, any files created by applications will also have different ownership in the file system.</p>
</blockquote>
<p>... this is a blessing and a curse, when using shared persistent volume claims for example (e.g. PVC's mounted in <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer"><code>ReadWriteMany</code></a> with multiple pods that read / write data - files created by one pod won't be accessible by the other pod because of the incorrect file ownership and permissions).</p>
<p>One way to get around this issue is using the <a href="https://docs.openshift.com/container-platform/4.9/authentication/managing-security-context-constraints.html" rel="nofollow noreferrer"><code>anyuid</code></a> security context which &quot;provides all features of the restricted SCC, but allows users to run with any UID and any GID&quot;.</p>
<p>When using the <code>anyuid</code> security context, we know the user and group ID's the pod(s) are going to run as, and we can set the permissions on the shared volume in advance. For example, where all pods run with the <code>restricted</code> security context by default:</p>
<p><a href="https://i.stack.imgur.com/aVeaI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/aVeaI.png" alt="restricted security context" /></a></p>
<p>When running the pod with the <code>anyuid</code> security context, OpenShift doesn't assign an arbitrary UID from the range of UID's allocated for the namespace:</p>
<p><a href="https://i.stack.imgur.com/VVTVN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VVTVN.png" alt="anyuid security context" /></a></p>
<p>This is just for example, but an image that is built with a non-root user with a fixed UID and GID (e.g. <code>1000:1000</code>) would run in OpenShift as that user, files would be created with the ownership of that user (e.g. <code>1000:1000</code>), permissions can be set on the PVC to the known UID and GID of the user set to run the service. For example, we can create a new PVC:</p>
<pre class="lang-shell prettyprint-override"><code>cat &lt;&lt;EOF |kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: k8s
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 8Gi
  storageClassName: portworx-shared-sc
EOF
</code></pre>
<p>... then mount it in a pod:</p>
<pre class="lang-shell prettyprint-override"><code>kubectl run -i --rm --tty ansible --image=lazybit/ansible:v4.0.0 --restart=Never -n k8s --overrides='
{
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;kind&quot;: &quot;Pod&quot;,
  &quot;spec&quot;: {
    &quot;serviceAccountName&quot;: &quot;default&quot;,
    &quot;containers&quot;: [
      {
        &quot;name&quot;: &quot;nginx&quot;,
        &quot;imagePullPolicy&quot;: &quot;Always&quot;,
        &quot;image&quot;: &quot;lazybit/ansible:v4.0.0&quot;,
        &quot;command&quot;: [&quot;ash&quot;],
        &quot;stdin&quot;: true,
        &quot;stdinOnce&quot;: true,
        &quot;tty&quot;: true,
        &quot;env&quot;: [
          {
            &quot;name&quot;: &quot;POD_NAME&quot;,
            &quot;valueFrom&quot;: {
              &quot;fieldRef&quot;: {
                &quot;apiVersion&quot;: &quot;v1&quot;,
                &quot;fieldPath&quot;: &quot;metadata.name&quot;
              }
            }
          }
        ],
        &quot;volumeMounts&quot;: [
          {
            &quot;mountPath&quot;: &quot;/data&quot;,
            &quot;name&quot;: &quot;data&quot;
          }
        ]
      }
    ],
    &quot;volumes&quot;: [
      {
        &quot;name&quot;: &quot;data&quot;,
        &quot;persistentVolumeClaim&quot;: {
          &quot;claimName&quot;: &quot;data&quot;
        }
      }
    ]
  }
}'
</code></pre>
<p>... and create files in the PVC as the <a href="https://github.com/lazybit-ch/ansible/blob/master/Dockerfile#L30" rel="nofollow noreferrer"><code>USER</code> set in the Dockerfile</a>.</p>
<p><a href="https://i.stack.imgur.com/cJCCW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cJCCW.png" alt="example file created as anyuid USER" /></a></p>
