<p>In my Kubernetes cluster, Rancher never creates Persistent Volumes after creating a Persistent Volume Claim and applying a Pod.</p>
<p><strong>Solution/Work around</strong> available under second update.</p>
<p>The cluster has been installed with Kubespray. The configuration for local path provisioning in <code>inventory/myclster/group_vars/k8s-cluster/addons.yml</code>:</p>
<pre><code># Rancher Local Path Provisioner
local_path_provisioner_enabled: true
# local_path_provisioner_namespace: &quot;local-path-storage&quot;
# local_path_provisioner_storage_class: &quot;local-path&quot;
# local_path_provisioner_reclaim_policy: Delete
# local_path_provisioner_claim_root: /opt/local-path-provisioner/
# local_path_provisioner_debug: false
# local_path_provisioner_image_repo: &quot;rancher/local-path-provisioner&quot;
# local_path_provisioner_image_tag: &quot;v0.0.14&quot;
# local_path_provisioner_helper_image_repo: &quot;busybox&quot;
# local_path_provisioner_helper_image_tag: &quot;latest&quot;

# Local volume provisioner deployment
local_volume_provisioner_enabled: false
# local_volume_provisioner_namespace: kube-system
# local_volume_provisioner_nodelabels:
#   - kubernetes.io/hostname
#   - topology.kubernetes.io/region
#   - topology.kubernetes.io/zone
# local_volume_provisioner_storage_classes:
#   local-storage:
#     host_dir: /mnt/disks
#     mount_dir: /mnt/disks
#     volume_mode: Filesystem
#     fs_type: ext4
#   fast-disks:
#     host_dir: /mnt/fast-disks
#     mount_dir: /mnt/fast-disks
#     block_cleaner_command:
#       - &quot;/scripts/shred.sh&quot;
#       - &quot;2&quot;
#     volume_mode: Filesystem
#     fs_type: ext4
</code></pre>
<p>Steps to recreate the problem:</p>
<p>Create PVC:<br />
<code>kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml</code></p>
<p>Result:
Created PVC, name <code>local-path-pvc</code>, status Pending, storage class <code>local-path</code></p>
<p>Create Pod<br />
<code>kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml</code></p>
<p>Result:<br />
Created Pod, name <code>create-pvc-123</code>, status Waiting:ContainerCreating.</p>
<p>Describing Pod with <code>kubectl</code>:</p>
<pre><code>Events:  
  Type     Reason            Age        From  Message
  ----     ------            ----       ----  -------
  Warning  FailedScheduling  &lt;unknown&gt;        error while running &quot;VolumeBinding&quot; prebind plugin for pod &quot;create-pvc-123&quot;: Failed to bind volumes: timed out waiting for the condition
</code></pre>
<p>I have tried different charts, and in all cases no Persistent Volume has been created.  The ServiceAccount local-path-provisioner-service-account exists. The Deployment of the local path provisioner has one Pod.</p>
<p><strong>UPDATE</strong><br />
On the server the logs contains several errors, <code>sudo journalctl -xeu kubelet | grep 'fail'</code>:</p>
<pre><code>...
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.000246  274306 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script podName:71b44438-fadb-4859-a788-8d911dfab2db nodeName:}&quot; failed. No retries permitted until 2020-10-12 16:54:40.000164134 +0200 CEST m=+9380.643933974 (durationBeforeRetry 1m4s). Error: &quot;MountVolume.SetUp failed for volume \&quot;script\&quot; (UniqueName: \&quot;kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script\&quot;) pod \&quot;create-pvc-80d115d9-98fd-4fcd-9e41-55b74f809efb\&quot; (UID: \&quot;71b44438-fadb-4859-a788-8d911dfab2db\&quot;) : configmap references non-existent config key: setup&quot;
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.404015  274306 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}&quot; failed. No retries permitted until 2020-10-12 16:54:40.403980548 +0200 CEST m=+9381.047750378 (durationBeforeRetry 1m4s). Error: &quot;MountVolume.SetUp failed for volume \&quot;script\&quot; (UniqueName: \&quot;kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\&quot;) pod \&quot;create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\&quot; (UID: \&quot;424b196e-5132-479a-8b95-63e41e0ea124\&quot;) : configmap references non-existent config key: setup&quot;
Oct 12 16:54:40 node1 kubelet[274306]: E1012 16:54:40.464999  274306 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}&quot; failed. No retries permitted until 2020-10-12 16:56:42.464936126 +0200 CEST m=+9503.108706016 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;script\&quot; (UniqueName: \&quot;kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\&quot;) pod \&quot;create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\&quot; (UID: \&quot;424b196e-5132-479a-8b95-63e41e0ea124\&quot;) : configmap references non-existent config key: setup&quot;

</code></pre>
<p><strong>UPDATE - solution?</strong><br />
I changed the ConfigMap 'local-path-config' as described in the <a href="https://github.com/rancher/local-path-provisioner#configuration" rel="nofollow noreferrer">docs</a>.<br />
However, the <a href="https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubernetes-apps/external_provisioner/local_path_provisioner/templates/local-path-storage-cm.yml.j2" rel="nofollow noreferrer">jinja template</a> in Kubespray lacks the properties 'setup' and 'teardown' in the configuration.</p>
<p>When I added de <code>setup</code> and <code>teardown</code> properties Kubernetes created the PV and the Pod started.</p>
<p><strong>What's the reason Kubespray doesn't provide these properties in the template?</strong></p>

erwineberhard
<p>Default version that kubespray should use is local_path_provisioner_image_tag: &quot;v0.0.14&quot;.
I suppose the source for the template is <a href="https://github.com/rancher/local-path-provisioner/blob/v0.0.14/deploy/example-config.yaml" rel="nofollow noreferrer">https://github.com/rancher/local-path-provisioner/blob/v0.0.14/deploy/example-config.yaml</a>.
It doesn't have setup and teardown properties, they were introduced in v0.0.15.</p>
