{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71829","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71829/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71829/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71829/events","html_url":"https://github.com/kubernetes/kubernetes/issues/71829","id":388533643,"node_id":"MDU6SXNzdWUzODg1MzM2NDM=","number":71829,"title":"vSphere cloud provider unable to detachVolume if node is deleted but not vSphere vm","user":{"login":"maplain","id":2901728,"node_id":"MDQ6VXNlcjI5MDE3Mjg=","avatar_url":"https://avatars2.githubusercontent.com/u/2901728?v=4","gravatar_id":"","url":"https://api.github.com/users/maplain","html_url":"https://github.com/maplain","followers_url":"https://api.github.com/users/maplain/followers","following_url":"https://api.github.com/users/maplain/following{/other_user}","gists_url":"https://api.github.com/users/maplain/gists{/gist_id}","starred_url":"https://api.github.com/users/maplain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maplain/subscriptions","organizations_url":"https://api.github.com/users/maplain/orgs","repos_url":"https://api.github.com/users/maplain/repos","events_url":"https://api.github.com/users/maplain/events{/privacy}","received_events_url":"https://api.github.com/users/maplain/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":874113778,"node_id":"MDU6TGFiZWw4NzQxMTM3Nzg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/vmware","name":"sig/vmware","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-12-07T07:16:09Z","updated_at":"2018-12-07T07:20:00Z","closed_at":null,"author_association":"NONE","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->\r\n/kind bug\r\n\r\n**What happened**:\r\nWe had a pod with persistent storage. We drained the node firstly. Then, we deleted the node through kubectl, but the worker vm keeps there. The PV was not attached to the node that the Pod migrated too.  As a result, rescheduled statefulset pod is stuck in pending state forever.\r\n\r\n**What you expected to happen**:\r\nPV is detached from old vm successfully so that statefulset pod can be recreated on another host without pending forever.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n1. deploy storage-class by this yaml\r\n```\r\nkind: StorageClass\r\napiVersion: storage.k8s.io/v1\r\nmetadata:\r\n  name: ci-storage\r\nprovisioner: kubernetes.io/vsphere-volume\r\nparameters:\r\n  diskformat: zeroedthick\r\n```\r\n2. deploy stateful-set by this yaml\r\n```\r\napiVersion: apps/v1beta1\r\nkind: StatefulSet\r\nmetadata:\r\n  name: sample-statefulset\r\nspec:\r\n  serviceName: \"pv-ex-svc\"\r\n  replicas: 3\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: pv-ex\r\n    spec:\r\n      terminationGracePeriodSeconds: 10\r\n      containers:\r\n      - name: pv-ex\r\n        image: ${PATH_TO_STATEFUL_SET}\r\n        ports:\r\n        - containerPort: 3000\r\n        volumeMounts:\r\n        - name: www\r\n          mountPath: /usr/share/pv-ex\r\n        env:\r\n        - name: LOG_FILE\r\n          value: /usr/share/pv-ex/log.txt\r\n  updateStrategy:\r\n    type: RollingUpdate\r\n  volumeClaimTemplates:\r\n  - metadata:\r\n      name: www\r\n    spec:\r\n      accessModes: [ \"ReadWriteOnce\" ]\r\n      storageClassName: ci-storage\r\n      resources:\r\n        requests:\r\n          storage: 1Gi\r\n```\r\n3. run \r\n```\r\n  kubectl --kubeconfig xxx drain \"${node_name}\" --grace-period 10 --force --delete-local-data  --ignore-daemonsets\r\n```\r\n4. run\r\n```\r\nkubectl --kubeconfig xxx delete node \"${node_name}\" --ignore-not-found\r\n```\r\n5. wait for 6 minutes\r\n6. have following logs in kube-controller-manager:\r\n```\r\nI1207 04:12:50.864000       1 node_status_updater.go:68] Could not update node status. Failed to find node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" in NodeInformer cache. Error: 'node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" not found'\r\nW1207 04:12:50.864336       1 reconciler.go:231] attacherDetacher.DetachVolume started for volume \"pvc-135ea502-f926-11e8-b7a5-005056a39e82\" (UniqueName: \"kubernetes.io/vsphere-volume/[iscsi-ds-0] kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\") on node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching\r\nE1207 04:12:50.864403       1 nodemanager.go:282] Error \"No VM found\" node info for node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" not found\r\nE1207 04:12:50.864715       1 vsphere.go:550] Cannot find node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" in cache. Node not found!!!\r\nE1207 04:12:50.864983       1 attacher.go:259] Error checking if volume (\"[iscsi-ds-0] kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\") is already attached to current node (\"997186e0-4c8c-42ba-8bc0-624d67d76c13\"). Will continue and try detach anyway. err=No VM found\r\nE1207 04:12:50.865103       1 nodemanager.go:282] Error \"No VM found\" node info for node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" not found\r\nE1207 04:12:50.865161       1 vsphere.go:550] Cannot find node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" in cache. Node not found!!!\r\nI1207 04:12:50.865417       1 vsphere.go:869] Node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\" does not exist, disk [iscsi-ds-0] kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk is already detached from node.\r\nI1207 04:12:50.865592       1 operation_generator.go:419] DetachVolume.Detach succeeded for volume \"pvc-135ea502-f926-11e8-b7a5-005056a39e82\" (UniqueName: \"kubernetes.io/vsphere-volume/[iscsi-ds-0] kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\") on node \"997186e0-4c8c-42ba-8bc0-624d67d76c13\"\r\nE1207 04:13:13.760273       1 virtualmachine.go:137] Failed to attach the disk with storagePolicy: \"\" on VM: \"\". err - Failed to add disk 'scsi1:0'.\r\n```\r\n\r\n**Anything else we need to know?**:\r\nIt's caused by:\r\n1. vcp removes node from cache in face of a node delete event: \r\n https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go#L1295\r\n2. vcp depends on the cache to find node in DetachDisk method:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go#L910\r\n3. if there is no node information in cache, vcp doesn't even check with vCenter to update cache, but directly asserts node is gone\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/nodemanager.go#L232\r\n4. vcp assumes if node could not be found, disk is detached successfully in DetachDisk:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go#L202\r\n\r\nTherefore, even though vm is still on vCenter with PV attached, DetachDisk could not successfully finish its job because of a cache miss. And once DetachDisk is called, no matter it's invoked by attachdetach.reconciler forcifully because of timeout, or following normal flow, it'll fool controller into thinking PV is detached successfully. This results in a pending stateful-set pod on another host.\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n```\r\nkubo@jumper:~$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"13\", GitVersion:\"v1.13.0\", GitCommit:\"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\", GitTreeState:\"clean\", BuildDate:\"2018-12-03T21:04:45Z\", GoVersion:\"go1.11.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.5\", GitCommit:\"753b2dbc622f5cc417845f0ff8a77f539a4213ea\", GitTreeState:\"clean\", BuildDate:\"2018-11-26T14:31:35Z\", GoVersion:\"go1.10.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n- Cloud provider or hardware configuration:\r\nvSphere\r\n- OS (e.g. from /etc/os-release):\r\n```\r\nNAME=\"Ubuntu\"\r\nVERSION=\"16.04.5 LTS (Xenial Xerus)\"\r\nID=ubuntu\r\nID_LIKE=debian\r\nPRETTY_NAME=\"Ubuntu 16.04.5 LTS\"\r\nVERSION_ID=\"16.04\"\r\nHOME_URL=\"http://www.ubuntu.com/\"\r\nSUPPORT_URL=\"http://help.ubuntu.com/\"\r\nBUG_REPORT_URL=\"http://bugs.launchpad.net/ubuntu/\"\r\nVERSION_CODENAME=xenial\r\nUBUNTU_CODENAME=xenial\r\n```\r\n- Kernel (e.g. `uname -a`):\r\n```\r\nLinux 56870dca-33af-48ca-919e-3bec31e14ed4 4.15.0-39-generic #42~16.04.1-Ubuntu SMP Wed Oct 24 17:09:54 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n- Install tools: PKS\r\n- Others:\r\n\r\n<!-- DO NOT EDIT BELOW THIS LINE -->\r\n/kind bug"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/445145509","html_url":"https://github.com/kubernetes/kubernetes/issues/71829#issuecomment-445145509","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71829","id":445145509,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTE0NTUwOQ==","user":{"login":"maplain","id":2901728,"node_id":"MDQ6VXNlcjI5MDE3Mjg=","avatar_url":"https://avatars2.githubusercontent.com/u/2901728?v=4","gravatar_id":"","url":"https://api.github.com/users/maplain","html_url":"https://github.com/maplain","followers_url":"https://api.github.com/users/maplain/followers","following_url":"https://api.github.com/users/maplain/following{/other_user}","gists_url":"https://api.github.com/users/maplain/gists{/gist_id}","starred_url":"https://api.github.com/users/maplain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maplain/subscriptions","organizations_url":"https://api.github.com/users/maplain/orgs","repos_url":"https://api.github.com/users/maplain/repos","events_url":"https://api.github.com/users/maplain/events{/privacy}","received_events_url":"https://api.github.com/users/maplain/received_events","type":"User","site_admin":false},"created_at":"2018-12-07T07:20:00Z","updated_at":"2018-12-07T07:20:00Z","author_association":"NONE","body":"@SandeepPissay @abrarshivani @divyenpatel "}}
{"actor":{"display_login":"maplain"}}