{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30067","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30067/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30067/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30067/events","html_url":"https://github.com/kubernetes/kubernetes/issues/30067","id":169350253,"number":30067,"title":"Cleanup orphaned pods (unmount pods directories)","user":{"login":"suquant","id":735363,"avatar_url":"https://avatars.githubusercontent.com/u/735363?v=3","gravatar_id":"","url":"https://api.github.com/users/suquant","html_url":"https://github.com/suquant","followers_url":"https://api.github.com/users/suquant/followers","following_url":"https://api.github.com/users/suquant/following{/other_user}","gists_url":"https://api.github.com/users/suquant/gists{/gist_id}","starred_url":"https://api.github.com/users/suquant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suquant/subscriptions","organizations_url":"https://api.github.com/users/suquant/orgs","repos_url":"https://api.github.com/users/suquant/repos","events_url":"https://api.github.com/users/suquant/events{/privacy}","received_events_url":"https://api.github.com/users/suquant/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/storage","name":"area/storage","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-04T11:14:03Z","updated_at":"2016-08-16T19:00:21Z","closed_at":null,"body":"Hi all,\r\n\r\nSometimes we delete/kill pods who is uses NFS directories which also run in kubernetes cluster or somewhere else.\r\n\r\n**Simple case**:\r\n 1. Run gluster server pod\r\n 2. Run app server which use gluster server volume as NFS mount point\r\n 3. Delete gluster pod\r\n 4. Delete app pod\r\n\r\n\r\ngluster.yaml\r\n\r\n```\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: gluster1\r\n  namespace: gitlab\r\n  labels:\r\n    component: glusterfs-storage\r\n    host: st01\r\nspec:\r\n  nodeSelector:\r\n    host: st01\r\n  containers:\r\n    - name: glusterfs-server\r\n      image: suquant/glusterd:3.6.kube\r\n      imagePullPolicy: Always\r\n      command:\r\n        - /kubernetes-glusterd\r\n      args:\r\n        - --namespace\r\n        - gitlab\r\n        - --labels\r\n        - component=glusterfs-storage\r\n      ports:\r\n        - containerPort: 24007\r\n        - containerPort: 24008\r\n        - containerPort: 49152\r\n        - containerPort: 38465\r\n        - containerPort: 38466\r\n        - containerPort: 38467\r\n        - containerPort: 2049\r\n        - containerPort: 111\r\n        - containerPort: 111\r\n          protocol: UDP\r\n      volumeMounts:\r\n        - name: brick\r\n          mountPath: /mnt/brick\r\n        - name: fuse\r\n          mountPath: /dev/fuse\r\n        - name: data\r\n          mountPath: /var/lib/glusterd\r\n      securityContext:\r\n        capabilities:\r\n          add:\r\n            - SYS_ADMIN\r\n            - MKNOD\r\n  volumes:\r\n    - name: brick\r\n      hostPath:\r\n        path: /mnt/gitlab\r\n    - name: fuse\r\n      hostPath:\r\n        path: /dev/fuse\r\n    - name: data\r\n      hostPath:\r\n        path: /opt/var/lib/gluster/gitlab\r\n```\r\n\r\napp.yaml\r\n\r\n```\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  labels:\r\n    name: gitlab\r\n    version: 8.10.3-ce.0\r\n  name: gitlab\r\n  namespace: gitlab\r\nspec:\r\n  nodeSelector:\r\n    host: app02\r\n  containers:\r\n    - name: gitlab\r\n      image: gitlab/gitlab-ce:8.10.3-ce.0\r\n      ports:\r\n        - name: ssh\r\n          containerPort: 22\r\n        - name: http\r\n          containerPort: 80\r\n      volumeMounts:\r\n        - name: data\r\n          mountPath: /var/opt/gitlab\r\n  volumes:\r\n    - name: data\r\n      nfs:\r\n        server: glusterfs-storage\r\n        path: /media\r\n\r\n```\r\n\r\nAfter delete/kill glusterfs pod before delete/kill app pod, which is gitlab app.\r\nWhat is happen in this case:\r\n  1. After delete/kill glusterfs server nfs was unavailable\r\n  2. gitlab also used nfs from glusterfs-storage which is unavailable after\r\n  3. After delete  app pod, cleanup method in kubernetes working incorrectly because unmount unused pod directory stay in NFS I/O wait which is unavailable forever.\r\n\r\nHow this appears in logs:\r\n\r\n```\r\n...\r\nAug 04 10:34:03 app03 kubelet-wrapper[15357]: E0804 10:34:03.552870   15357 kubelet.go:2596] Failed cleaning pods: remove /var/lib/kubelet/pods/dc8634a9-56fd-11e6-b43a-305a3a75dc73/volumes/kubernetes.io~nfs/image-store: device or resource busy\r\n...\r\n```\r\n\r\n**Solutions:**\r\n\r\nWe need detect network storage mount point which is NFS one of that and force unmount it with --force and --lazy options.\r\n\r\n```\r\numount --force --lazy /var/lib/kubelet/pods/dc8634a9-56fd-11e6-b43a-305a3a75dc73/volumes/kubernetes.io~nfs/image-store\r\n```\r\n\r\nAlso this kind of problem appears in secrets (http://kubernetes.io/docs/user-guide/secrets/) and configmaps (http://kubernetes.io/docs/user-guide/configmap/) mount points.\r\n\r\nThank you."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/240203490","html_url":"https://github.com/kubernetes/kubernetes/issues/30067#issuecomment-240203490","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30067","id":240203490,"user":{"login":"jeffvance","id":3076896,"avatar_url":"https://avatars.githubusercontent.com/u/3076896?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffvance","html_url":"https://github.com/jeffvance","followers_url":"https://api.github.com/users/jeffvance/followers","following_url":"https://api.github.com/users/jeffvance/following{/other_user}","gists_url":"https://api.github.com/users/jeffvance/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffvance/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffvance/subscriptions","organizations_url":"https://api.github.com/users/jeffvance/orgs","repos_url":"https://api.github.com/users/jeffvance/repos","events_url":"https://api.github.com/users/jeffvance/events{/privacy}","received_events_url":"https://api.github.com/users/jeffvance/received_events","type":"User","site_admin":false},"created_at":"2016-08-16T19:00:21Z","updated_at":"2016-08-16T19:00:21Z","body":"@jingxu97, the Name() method is called to construct the path for mounting and unmounting the vol. The mount was always done on the inner spec name (correct). But sometimes the umount was attempted against the outer spec name. This seemed to me to be a timing issue but not confirmed. A way that outer spec name could be used instead of inner name is when the Name() method (https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/plugins.go#L233) sees that spec.Volume is nil and therefore returns PersistentVolume.Name rather than Volume.Name. I was not able to debug this further."}}
{"actor":{"display_login":"jeffvance"}}