<p>I'm stuck with an annoying issue, where my pod can't access the mounted persistent volume.</p>
<p><strong>Kubeadm</strong>: <code>v1.19.2</code><br />
<strong>Docker</strong>: <code>19.03.13</code><br />
<strong>Zookeeper image</strong>: <code>library/zookeeper:3.6</code><br />
<strong>Cluster info</strong>: <code>Locally hosted, no Cloud Provide</code></p>
<p>K8s configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  selector:
    app: zk
  ports:
    - port: 2888
      targetPort: 2888
      name: server
      protocol: TCP
    - port: 3888
      targetPort: 3888
      name: leader-election
      protocol: TCP
  clusterIP: &quot;&quot;
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  selector:
    app: zk
  ports:
    - name: client
      protocol: TCP
      port: 2181
      targetPort: 2181
  type: LoadBalancer
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: zk
    spec:
      volumes:
        - name: zoo-config
          configMap:
            name: zoo-config
        - name: datadir
          persistentVolumeClaim:
            claimName: zoo-pvc
      containers:
        - name: zookeeper
          imagePullPolicy: Always
          image: &quot;library/zookeeper:3.6&quot;
          resources:
            requests:
              memory: &quot;1Gi&quot;
              cpu: &quot;0.5&quot;
          ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: server
            - containerPort: 3888
              name: leader-election
          volumeMounts:
            - name: datadir
              mountPath: /var/lib/zookeeper/data
            - name: zoo-config
              mountPath: /conf
      securityContext:
        fsGroup: 2000
        runAsUser: 1000
        runAsNonRoot: true
  volumeClaimTemplates:
    - metadata:
        name: datadir
        annotations:
          volume.beta.kubernetes.io/storage-class: local-storage
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        storageClassName: local-storage
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zoo-config
  namespace: default
data:
  zoo.cfg: |
    tickTime=10000
    dataDir=/var/lib/zookeeper/data
    clientPort=2181
    initLimit=10
    syncLimit=4
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: zoo-pv
  labels:
    type: local
spec:
  storageClassName: local-storage
  persistentVolumeReclaimPolicy: Retain
  hostPath:
      path: &quot;/mnt/data&quot;
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - &lt;node-name&gt;

</code></pre>
<p>I've tried running the pod as root with the following security context, which I know is a terrible idea, purely as a test. This however caused a bunch of other issues.</p>
<pre class="lang-yaml prettyprint-override"><code>securityContext:
  fsGroup: 0
  runAsUser: 0
</code></pre>
<p>Once the pod starts up the logs contain the following,</p>
<pre><code>Zookeeper JMX enabled by default
Using config: /conf/zoo.cfg
&lt;log4j Warnings&gt;
Unable too access datadir, exiting abnormally
</code></pre>
<p>Inspecting the pod, provides me with the following information,</p>
<pre><code>~$ kubectl describe pod/zk-0
Name:         zk-0
Namespace:    default
Priority:     0
Node:         &lt;node&gt;
Start Time:   Sat, 26 Sep 2020 15:48:00 +0200
Labels:       app=zk
              controller-revision-hash=zk-6c68989bd
              statefulset.kubernetes.io/pod-name=zk-0
Annotations:  &lt;none&gt;
Status:       Running
IP:           &lt;IP&gt;
IPs:
  IP:           &lt;IP&gt;
Controlled By:  StatefulSet/zk
Containers:
  zookeeper:
    Container ID:   docker://281e177d677394604785542c231d21b71f1666a22e74c1c10ef88491dad7a522
    Image:          library/zookeeper:3.6
    Image ID:       docker-pullable://zookeeper@sha256:6c051390cfae7958ff427834937c353fc6c34484f6a84b3e4bc8c512b53a16f6
    Ports:          2181/TCP, 2888/TCP, 3888/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Sat, 26 Sep 2020 16:04:26 +0200
      Finished:     Sat, 26 Sep 2020 16:04:27 +0200
    Ready:          False
    Restart Count:  8
    Requests:
      cpu:        500m
      memory:     1Gi
    Environment:  &lt;none&gt;
    Mounts:
      /conf from zoo-config (rw)
      /var/lib/zookeeper/data from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-88x56 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-zk-0
    ReadOnly:   false
  zoo-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      zoo-config
    Optional:  false
  default-token-88x56:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-88x56
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned default/zk-0 to &lt;node&gt;
  Normal   Pulled     17m                   kubelet            Successfully pulled image &quot;library/zookeeper:3.6&quot; in 1.932381527s
  Normal   Pulled     17m                   kubelet            Successfully pulled image &quot;library/zookeeper:3.6&quot; in 1.960610662s
  Normal   Pulled     17m                   kubelet            Successfully pulled image &quot;library/zookeeper:3.6&quot; in 1.959935633s
  Normal   Created    16m (x4 over 17m)     kubelet            Created container zookeeper
  Normal   Pulled     16m                   kubelet            Successfully pulled image &quot;library/zookeeper:3.6&quot; in 1.92551645s
  Normal   Started    16m (x4 over 17m)     kubelet            Started container zookeeper
  Normal   Pulling    15m (x5 over 17m)     kubelet            Pulling image &quot;library/zookeeper:3.6&quot;
  Warning  BackOff    2m35s (x71 over 17m)  kubelet            Back-off restarting failed container

</code></pre>
<p>To me, it seems like the pod has full <code>rw</code> access to the volume, so I'm unsure why it's still refusing to access the directory. Any help will be appreciated!</p>

<p>After quite some digging, I finally figured out why it wasn't working. The logs were actually telling me all I needed to know in the end, the mounted <code>persistentVolumeClaim</code> simply did not have the correct file permissions to read from the mounted hostpath <code>/mnt/data</code> directory</p>
<p>To fix this, in a somewhat <em>hacky</em> way, I gave <code>read</code>, <code>write</code> &amp; <code>execute</code> permissions to all.</p>
<pre><code>chmod 777 /mnt/data
</code></pre>
<p>Overview can be found <a href="https://www.pluralsight.com/blog/it-ops/linux-file-permissions" rel="nofollow noreferrer">here</a></p>
<p>This is definitely not the most secure way, of fixing the issue, and I would strongly advise against using this in any production like environment.</p>
<p>Probably a better approach would be the following</p>
<pre><code>sudo usermod -a -G 1000 1000
</code></pre>
