<p>I have a 3 node kubernetes cluster installed using kubeadm. However, when pods are scheduled on a particular kubernetes worker, it fails to do mounting and sometimes goes to pending state altogether. Can someone  please suggest as to how to debug this.</p>
<p>Update -:
Error log of worker.</p>
<pre><code>Jun 24 19:14:19 kub-minion-1 kubelet[3268]: E0624 19:14:19.481535    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:21 kub-minion-1 kubelet[3268]: E0624 19:14:21.489485    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:22 kub-minion-1 kubelet[3268]: E0624 19:14:22.446385    3268 pod_workers.go:191] Error syncing pod 871b0139-85b7-46c0-8616-24eb87e2d38d (&quot;calico-kube-controllers-76d4774d89-88djk_kube-system(871b0139-85b7-46c0-8616-24eb87e2d38d)&quot;), skipping: failed to &quot;StartContainer&quot; for &quot;calico-kube-controllers&quot; with ImagePullBackOff: &quot;Back-off pulling image \&quot;calico/kube-controllers:v3.14.1\&quot;&quot;
Jun 24 19:14:23 kub-minion-1 kubelet[3268]: E0624 19:14:23.483911    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:25 kub-minion-1 kubelet[3268]: E0624 19:14:25.494279    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:27 kub-minion-1 kubelet[3268]: E0624 19:14:27.519243    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:29 kub-minion-1 kubelet[3268]: E0624 19:14:29.505746    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:31 kub-minion-1 kubelet[3268]: E0624 19:14:31.555006    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.470044    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.609573    3268 configmap.go:200] Couldn't get configMap default/izac-cp-kafka-connect-jmx-configmap: object &quot;default&quot;/&quot;izac-cp-kafka-connect-jmx-configmap&quot; not registered
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.609639    3268 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/configmap/82f16d16-2363-4167-a7a8-6af0ce96ba95-jmx-config podName:82f16d16-2363-4167-a7a8-6af0ce96ba95 nodeName:}&quot; failed. No retries permitted until 2020-06-24 19:16:35.609618005 +0530 IST m=+571503.228144805 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;jmx-config\&quot; (UniqueName: \&quot;kubernetes.io/configmap/82f16d16-2363-4167-a7a8-6af0ce96ba95-jmx-config\&quot;) pod \&quot;izac-cp-kafka-connect-8c6d86c4d-r5r4s\&quot; (UID: \&quot;82f16d16-2363-4167-a7a8-6af0ce96ba95\&quot;) : object \&quot;default\&quot;/\&quot;izac-cp-kafka-connect-jmx-configmap\&quot; not registered&quot;
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.609853    3268 secret.go:195] Couldn't get secret default/nginx-ingress-backend-token-d6drl: object &quot;default&quot;/&quot;nginx-ingress-backend-token-d6drl&quot; not registered
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.609892    3268 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/secret/ac02ceb4-726d-4209-a97b-d3f4249d646a-nginx-ingress-backend-token-d6drl podName:ac02ceb4-726d-4209-a97b-d3f4249d646a nodeName:}&quot; failed. No retries permitted until 2020-06-24 19:16:35.609874166 +0530 IST m=+571503.228400966 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;nginx-ingress-backend-token-d6drl\&quot; (UniqueName: \&quot;kubernetes.io/secret/ac02ceb4-726d-4209-a97b-d3f4249d646a-nginx-ingress-backend-token-d6drl\&quot;) pod \&quot;nginx-ingress-default-backend-7c868597f4-f8rtt\&quot; (UID: \&quot;ac02ceb4-726d-4209-a97b-d3f4249d646a\&quot;) : object \&quot;default\&quot;/\&quot;nginx-ingress-backend-token-d6drl\&quot; not registered&quot;
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.610039    3268 configmap.go:200] Couldn't get configMap default/izac-cp-schema-registry-jmx-configmap: object &quot;default&quot;/&quot;izac-cp-schema-registry-jmx-configmap&quot; not registered
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.610072    3268 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/configmap/7bf54920-7557-4149-90da-3e8f261fc18c-jmx-config podName:7bf54920-7557-4149-90da-3e8f261fc18c nodeName:}&quot; failed. No retries permitted until 2020-06-24 19:16:35.610056758 +0530 IST m=+571503.228583538 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;jmx-config\&quot; (UniqueName: \&quot;kubernetes.io/configmap/7bf54920-7557-4149-90da-3e8f261fc18c-jmx-config\&quot;) pod \&quot;izac-cp-schema-registry-6cbf6d694b-6lwl6\&quot; (UID: \&quot;7bf54920-7557-4149-90da-3e8f261fc18c\&quot;) : object \&quot;default\&quot;/\&quot;izac-cp-schema-registry-jmx-configmap\&quot; not registered&quot;
Jun 24 19:14:33 kub-minion-1 kubelet[3268]: E0624 19:14:33.610198    3268 secret.go:195] Couldn't get secret default/izac-jobserver-token-qdfps: object &quot;default&quot;/&quot;izac-jobserver-token-qdfps&quot; not registered
Jun 24 19:14:36 kub-minion-1 kubelet[3268]: E0624 19:14:36.155302    3268 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/secret/a6125528-30b9-4da5-9449-ff32f17a43f7-izac-kafkaadmin-token-d5vjn podName:a6125528-30b9-4da5-9449-ff32f17a43f7 nodeName:}&quot; failed. No retries permitted until 2020-06-24 19:16:38.155279619 +0530 IST m=+571505.773806419 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;izac-kafkaadmin-token-d5vjn\&quot; (UniqueName: \&quot;kubernetes.io/secret/a6125528-30b9-4da5-9449-ff32f17a43f7-izac-kafkaadmin-token-d5vjn\&quot;) pod \&quot;izac-kafkaadmin-7db7645d4-n2nrk\&quot; (UID: \&quot;a6125528-30b9-4da5-9449-ff32f17a43f7\&quot;) : object \&quot;default\&quot;/\&quot;izac-kafkaadmin-token-d5vjn\&quot; not registered&quot;
Jun 24 19:14:36 kub-minion-1 kubelet[3268]: E0624 19:14:36.155353    3268 secret.go:195] Couldn't get secret default/izac-eventview-token-hx56g: object &quot;default&quot;/&quot;izac-eventview-token-hx56g&quot; not registered
Jun 24 19:14:36 kub-minion-1 kubelet[3268]: E0624 19:14:36.155386    3268 nestedpendingoperations.go:301] Operation for &quot;{volumeName:kubernetes.io/secret/f7aa6503-0424-406e-9f27-517c3046f38a-izac-eventview-token-hx56g podName:f7aa6503-0424-406e-9f27-517c3046f38a nodeName:}&quot; failed. No retries permitted until 2020-06-24 19:16:38.155370912 +0530 IST m=+571505.773897711 (durationBeforeRetry 2m2s). Error: &quot;MountVolume.SetUp failed for volume \&quot;izac-eventview-token-hx56g\&quot; (UniqueName: \&quot;kubernetes.io/secret/f7aa6503-0424-406e-9f27-517c3046f38a-izac-eventview-token-hx56g\&quot;) pod \&quot;izac-eventview-bbc689b4-8wzfd\&quot; (UID: \&quot;f7aa6503-0424-406e-9f27-517c3046f38a\&quot;) : object \&quot;default\&quot;/\&quot;izac-eventview-token-hx56g\&quot; not registered&quot;
Jun 24 19:14:37 kub-minion-1 kubelet[3268]: E0624 19:14:37.569743    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:39 kub-minion-1 kubelet[3268]: E0624 19:14:39.549682    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:41 kub-minion-1 kubelet[3268]: E0624 19:14:41.461248    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:43 kub-minion-1 kubelet[3268]: E0624 19:14:43.562688    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:45 kub-minion-1 kubelet[3268]: E0624 19:14:45.469403    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:47 kub-minion-1 kubelet[3268]: E0624 19:14:47.469884    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:48 kub-minion-1 kubelet[3268]: E0624 19:14:48.469826    3268 pod_workers.go:191] Error syncing pod 871b0139-85b7-46c0-8616-24eb87e2d38d (&quot;calico-kube-controllers-76d4774d89-88djk_kube-system(871b0139-85b7-46c0-8616-24eb87e2d38d)&quot;), skipping: failed to &quot;StartContainer&quot; for &quot;calico-kube-controllers&quot; with ImagePullBackOff: &quot;Back-off pulling image \&quot;calico/kube-controllers:v3.14.1\&quot;&quot;
Jun 24 19:14:49 kub-minion-1 kubelet[3268]: E0624 19:14:49.466371    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:51 kub-minion-1 kubelet[3268]: E0624 19:14:51.559734    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:53 kub-minion-1 kubelet[3268]: E0624 19:14:53.584893    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:55 kub-minion-1 kubelet[3268]: E0624 19:14:55.445726    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:57 kub-minion-1 kubelet[3268]: E0624 19:14:57.609778    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:14:59 kub-minion-1 kubelet[3268]: E0624 19:14:59.498661    3268 kubelet_volumes.go:154] orphaned pod &quot;093642bc-5ef9-4c41-b9f4-032a6866b400&quot; found, but volume paths are still present on disk : There were a total of 38 errors similar to this. Turn up verbosity to see them.
Jun 24 19:15:00 kub-minion-1 kubelet[3268]: E0624 19:15:00.445806    3268 pod_workers.go:191] Error syncing pod 871b0139-85b7-46c0-8616-24eb87e2d38d (&quot;calico-kube-controllers-76d4774d89-88djk_kube-system(871b0139-85b7-46c0-8616-24eb87e2d38d)&quot;), skipping: failed to &quot;StartContainer&quot; for &quot;calico-kube-controllers&quot; with ImagePullBackOff: &quot;Back-off pulling image \&quot;calico/kube-controllers:v3.14.1\&quot;&quot;

</code></pre>
<p>Update 2-:
Kubectl describe pod  gives -:</p>
<pre><code> Normal   Scheduled    25m                  default-scheduler      Successfully assigned default/izac-eventview-78775467d6-6bvxq to kub-minion-1
  Warning  FailedMount  2m1s (x16 over 24m)  kubelet, kub-minion-1  MountVolume.SetUp failed for volume &quot;izac-eventview-token-hrzcc&quot; : mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/6fd35008-2adf-4a35-a88a-28c2991d40ac/volumes/kubernetes.io~secret/izac-eventview-token-hrzcc --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/6fd35008-2adf-4a35-a88a-28c2991d40ac/volumes/kubernetes.io~secret/izac-eventview-token-hrzcc
Output: Failed to start transient scope unit: Connection timed out

</code></pre>

<p>The issue was that StorageOS pod was taking up certain OS specific service which was preventing mounting. I had to shutdown Kubelet Service forecibly, and delete StorageOS operator. Then had to restart the entire cluster after deleting pods in minion.</p>
