{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27058","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27058/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27058/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27058/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27058","id":159228197,"number":27058,"title":"rw gce pd mounted as ro","user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/storage","name":"area/storage","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.3","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18/labels","id":1638460,"number":18,"title":"v1.3","description":"https://github.com/kubernetes/kubernetes/wiki/Release-1.3","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":105,"closed_issues":718,"state":"open","created_at":"2016-03-11T07:40:20Z","updated_at":"2016-06-15T21:53:55Z","due_on":"2016-06-24T07:00:00Z","closed_at":null},"comments":7,"created_at":"2016-06-08T17:38:17Z","updated_at":"2016-06-15T21:53:55Z","closed_at":"2016-06-15T21:53:55Z","body":"I have a pvc like:\r\n```\r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n  annotations:\r\n    pv.kubernetes.io/bind-completed: \"yes\"\r\n    pv.kubernetes.io/bound-by-controller: \"yes\"\r\n    volume.alpha.kubernetes.io/storage-class: anything\r\n  creationTimestamp: 2016-06-08T07:06:29Z\r\n  labels:\r\n    app: zk\r\n  name: datadir-zoo-0\r\n  namespace: e2e-tests-petset-7zm2s\r\n  resourceVersion: \"182261\"\r\n  selfLink: /api/v1/namespaces/e2e-tests-petset-7zm2s/persistentvolumeclaims/datadir-zoo-0\r\n  uid: 85da26bf-2d47-11e6-8c15-42010af00002\r\nspec:\r\n  accessModes:\r\n  - ReadWriteOnce\r\n  resources:\r\n    requests:\r\n      storage: 1Gi\r\n  volumeName: pvc-85da26bf-2d47-11e6-8c15-42010af00002\r\nstatus:\r\n  phase: Bound\r\n```\r\n\r\nIt created a r/w pd (verified through the pantheon console) but the actual disk is mounted as ro.\r\n```\r\nroot@e2e-test-beeps-minion-group-nlnf:/var/lib/kubelet/pods/c86d8439-2d47-11e6-8c15-42010af00002/volumes# mount | grep pvc-85da26bf-2d47-11e6-8c15-42010af00002\r\n/dev/sdf on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/e2e-test-beeps-dynamic-pvc-85da26bf-2d47-11e6-8c15-42010af00002 type ext4 (ro,relatime,data=ordered)\r\n/dev/sdf on /var/lib/kubelet/pods/c86d8439-2d47-11e6-8c15-42010af00002/volumes/kubernetes.io~gce-pd/pvc-85da26bf-2d47-11e6-8c15-42010af00002 type ext4 (ro,relatime,data=ordered)\r\n\r\nroot@e2e-test-beeps-minion-group-nlnf:/var/lib/kubelet/pods/c86d8439-2d47-11e6-8c15-42010af00002/volumes# ls kubernetes.io~gce-pd/pvc-85da26bf-2d47-11e6-8c15-42010af00002/\r\nls: reading directory kubernetes.io~gce-pd/pvc-85da26bf-2d47-11e6-8c15-42010af00002/: Input/output error\r\n\r\n```\r\n\r\nSimilar disks created previously are mounted as r/w on the same vm.\r\n```\r\nroot@e2e-test-beeps-minion-group-nlnf:/var/lib/kubelet/pods/527ec9be-2d3f-11e6-8c15-42010af00002/volumes# mount -v | grep pvc-a418664d-2d38-11e6-8c15-42010af00002\r\n/dev/sde on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/e2e-test-beeps-dynamic-pvc-a418664d-2d38-11e6-8c15-42010af00002 type ext4 (rw,relatime,data=ordered)\r\n/dev/sde on /var/lib/kubelet/pods/527ec9be-2d3f-11e6-8c15-42010af00002/volumes/kubernetes.io~gce-pd/pvc-a418664d-2d38-11e6-8c15-42010af00002 type ext4 (rw,relatime,data=ordered)\r\n\r\n```\r\n\r\nOddly i couldn't find the partition identifier when i tried to remount it manually\r\n```\r\nroot@e2e-test-beeps-minion-group-nlnf:/var/lib/kubelet/pods/c86d8439-2d47-11e6-8c15-42010af00002/volumes/kubernetes.io~gce-pd# ls /dev/sdf\r\nls: cannot access /dev/sdf: No such file or directory\r\n\r\nroot@e2e-test-beeps-minion-group-nlnf:/var/lib/kubelet/pods/527ec9be-2d3f-11e6-8c15-42010af00002/volumes# ls /dev/sde\r\n/dev/sde\r\n```\r\n\r\nMy pod will keep crashlooping till the volume is r/w, how to recover (without deleting pod)?\r\n@kubernetes/sig-storage "}}
{"actor":{"display_login":"saad-ali"}}