<p>I'm trying to make a k8s pod able to use PMEM without using the <code>privileged mode</code>. The way I'm trying is to create a local PV on top of a fsdax directory with PVC in k8s and let my pod use it. However, I always get the <code>MountVolume.NewMounter initialization failed ... : path does not exist</code> error.</p>
<p>Here are my yaml files and PMEM status:</p>
<h3>Storage Class yaml:</h3>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<h3>PV yaml:</h3>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: pmem-pv-volume
spec:
  capacity:
    storage: 50Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/pmem0/vol1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values:
          - pmem
</code></pre>
<h3>PVC yaml:</h3>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-pv-claim
spec:
  storageClassName: local-storage
  volumeName: pmem-pv-volume
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
</code></pre>
<h3>Pod yaml:</h3>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: daemon
  labels:
    env: test
spec:
  hostNetwork: true
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - pmem
  containers:
  - name: daemon-container
    command: [&quot;/usr/bin/bash&quot;, &quot;-c&quot;, &quot;sleep 3600&quot;]
    image: mm:v2
    imagePullPolicy: Never
    volumeMounts:
    - mountPath: /mnt/pmem
      name: pmem-pv-storage
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/log/memverge
      name: log
    - mountPath: /var/memverge/data
      name: data
  volumes:
    - name: pmem-pv-storage
      persistentVolumeClaim:
        claimName: pmem-pv-claim
    - name: tmp
      hostPath:
        path: /tmp
    - name: log
      hostPath:
        path: /var/log/memverge
    - name: data
      hostPath:
        path: /var/memverge/data
</code></pre>
<h3>Some status and k8s outputs:</h3>
<pre><code>$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 745.2G  0 disk
├─sda1        8:1    0     1G  0 part /boot
└─sda2        8:2    0   740G  0 part
  ├─cl-root 253:0    0   188G  0 lvm  /
  ├─cl-swap 253:1    0    32G  0 lvm  [SWAP]
  └─cl-home 253:2    0   520G  0 lvm  /home
sr0          11:0    1  1024M  0 rom
nvme0n1     259:0    0     7T  0 disk
└─nvme0n1p1 259:1    0     7T  0 part /mnt/nvme
pmem0       259:2    0 100.4G  0 disk /mnt/pmem0
</code></pre>
<pre><code>$ kubectl get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS    REASON   AGE
pmem-pv-volume   50Gi       RWO            Delete           Bound    default/pmem-pv-claim   local-storage            20h
</code></pre>
<pre><code>$ kubectl get pvc
NAME            STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pmem-pv-claim   Bound    pmem-pv-volume   50Gi       RWO            local-storage   20h
</code></pre>
<pre><code>$ kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
default       daemon                             0/1     ContainerCreating   0          20h
kube-system   coredns-74ff55c5b-5crgg            1/1     Running             0          20h
kube-system   etcd-minikube                      1/1     Running             0          20h
kube-system   kube-apiserver-minikube            1/1     Running             0          20h
kube-system   kube-controller-manager-minikube   1/1     Running             0          20h
kube-system   kube-proxy-2m7p6                   1/1     Running             0          20h
kube-system   kube-scheduler-minikube            1/1     Running             0          20h
kube-system   storage-provisioner                1/1     Running             0          20h
</code></pre>
<pre><code>$ kubectl get events
LAST SEEN   TYPE      REASON        OBJECT       MESSAGE
108s        Warning   FailedMount   pod/daemon   MountVolume.NewMounter initialization failed for volume &quot;pmem-pv-volume&quot; : path &quot;/mnt/pmem0/vol1&quot; does not exist
47m         Warning   FailedMount   pod/daemon   Unable to attach or mount volumes: unmounted volumes=[pmem-pv-storage], unattached volumes=[tmp log data default-token-4t8sv pmem-pv-storage]: timed out waiting for the condition
37m         Warning   FailedMount   pod/daemon   Unable to attach or mount volumes: unmounted volumes=[pmem-pv-storage], unattached volumes=[default-token-4t8sv pmem-pv-storage tmp log data]: timed out waiting for the condition
13m         Warning   FailedMount   pod/daemon   Unable to attach or mount volumes: unmounted volumes=[pmem-pv-storage], unattached volumes=[pmem-pv-storage tmp log data default-token-4t8sv]: timed out waiting for the condition
$ ls -l /mnt/pmem0
total 20
drwx------ 2 root root 16384 Jan 20 15:35 lost+found
drwxrwxrwx 2 root root  4096 Jan 21 17:56 vol1
</code></pre>
<p>It is complaining <code>path &quot;/mnt/pmem0/vol1&quot; does not exist</code> but it actually does exist:</p>
<pre><code>$ ls -l /mnt/pmem0
total 20
drwx------ 2 root root 16384 Jan 20 15:35 lost+found
drwxrwxrwx 2 root root  4096 Jan 21 17:56 vol1
</code></pre>
<p>Besides using local-PV, I also tried:</p>
<ol>
<li><p>PMEM-CSI. But the PMEM-CSI method is blocked from me by a containerd/kernel issue: <a href="https://github.com/containerd/containerd/issues/3221" rel="nofollow noreferrer">https://github.com/containerd/containerd/issues/3221</a></p>
</li>
<li><p>PV. When I trying to create PV backed by PMEM, the pod cannot claim the PMEM storage correctly, but always mount as an overlay fs which is on top of <code>/</code> on the host.</p>
</li>
</ol>
<p>Could anyone give some help? Thanks a lot!</p>

TOM ZHANG
<p>As was discussed in comments:</p>
<p>Using minikube, rancher and any other containerized version of kubelets will lead to <a href="https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/issues/107" rel="nofollow noreferrer">MountVolume.NewMounter initialization failed for volume</a>, stating this path <strong>does not</strong>  exist.</p>
<blockquote>
<p>If the kubelet is running in a container, it cannot access the host
filesystem at the same path. You must adjust hostDir to the correct
path in the kubelet container.</p>
</blockquote>
<p>Also what you can do is add bindings for local volumes as was <a href="https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/issues/107#issuecomment-498619999" rel="nofollow noreferrer">suggested on github</a>. Please adjust copy pasted example for your needs, if you will use it</p>
<pre><code>    &quot;HostConfig&quot;: {
        &quot;Binds&quot;: [
            &quot;/mnt/local:/mnt/local&quot;
        ],
</code></pre>
<p>Regular installations(not conteinerized), like <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/" rel="nofollow noreferrer">kubeadm</a> will not act the same and you will not receive such errors.</p>
