{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60298","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60298/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60298/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60298/events","html_url":"https://github.com/kubernetes/kubernetes/issues/60298","id":299692264,"number":60298,"title":"PVC status: \"Lost\" (AWS volumes)","user":{"login":"mthoretton","id":6221533,"avatar_url":"https://avatars0.githubusercontent.com/u/6221533?v=4","gravatar_id":"","url":"https://api.github.com/users/mthoretton","html_url":"https://github.com/mthoretton","followers_url":"https://api.github.com/users/mthoretton/followers","following_url":"https://api.github.com/users/mthoretton/following{/other_user}","gists_url":"https://api.github.com/users/mthoretton/gists{/gist_id}","starred_url":"https://api.github.com/users/mthoretton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mthoretton/subscriptions","organizations_url":"https://api.github.com/users/mthoretton/orgs","repos_url":"https://api.github.com/users/mthoretton/repos","events_url":"https://api.github.com/users/mthoretton/events{/privacy}","received_events_url":"https://api.github.com/users/mthoretton/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-02-23T12:12:46Z","updated_at":"2018-02-26T16:18:22Z","closed_at":null,"author_association":"NONE","body":"/kind bug\r\n\r\n**Problem**:\r\n\r\n`kubectl get pvc`\r\n![screen shot 2018-02-23 at 11 50 07](https://user-images.githubusercontent.com/6221533/36590685-be39844c-188f-11e8-9854-ba16ee9e0a0b.png)\r\n\r\nWe are running a kafka cluster in k8s. kafka-servers run as \"StatefulSet\". They're using PVC to store data:\r\n```\r\n# kafka manifest\r\n# ...\r\n  volumeClaimTemplates:\r\n  - metadata:\r\n      name: data\r\n    spec:\r\n      storageClassName: generic\r\n      accessModes: [ \"ReadWriteOnce\" ]\r\n      resources:\r\n        requests:\r\n          storage: 50Gi\r\n```\r\nThis was working pretty well.\r\nAt some point, all PVCs for the kafka StatefulSet got a \"Lost\" status. Running pod/containers are still bound to the AWS volumes though (=> we can still access those volumes mounted in pods filesystem). The main problem is, when we run an update for instance, the pods won't be able to start as the PVC state is not OK.\r\nkubectl is logging a few times/second this:\r\n```\r\nFeb 23 09:46:37 worker-generic3.xxx.com kubelet-wrapper[913]: E0223 09:46:37.509228     913 desired_state_of_world_populator.go:274] Error processing volume \"data\" for pod \"kafka-1_namespace(28f9e78e-165f-11e8-b0b4-02f219457d76)\": error processing PVC \"namespace\"/\"data-kafka-1\": PVC namespace/data-kafka-1 has non-bound phase (\"Lost\") or empty pvc.Spec.VolumeName (\"pvc-e50392dc-161f-11e8-b0b4-02f219457d76\")\r\n```\r\n\r\nAWS volumes are still running and healthy/with the right tags:\r\n![screen shot 2018-02-23 at 13 07 43](https://user-images.githubusercontent.com/6221533/36593593-a78b26f0-189a-11e8-8472-04a0ca3d4262.png)\r\n\r\n\r\nWe did check many things already but no way to find any explanation. Also we didn't find that much about what's the meaning of \"Lost\" status for PVCs.\r\nAs It's a sandbox cluster (we run manual commands and potentially broken configs) we could have broken something accidentally, but I don't see how it can lead to this situation.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\nNo idea. It's the first time we see that in a few months. Also, it only happened on one sandbox cluster so far.\r\n\r\n**Anything else we need to know?**:\r\n\r\nI don't think so but I'll provide everything needed to help resolving this issue. Thanks for looking into this :)\r\n\r\n---\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.3\", GitCommit:\"d2835416544f298c919e2ead3be3d0864b52323b\", GitTreeState:\"clean\", BuildDate:\"2018-02-09T21:51:54Z\", GoVersion:\"go1.9.4\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.5\", GitCommit:\"cce11c6a185279d037023e02ac5249e14daa22bf\", GitTreeState:\"clean\", BuildDate:\"2017-12-07T16:05:18Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n- Cloud provider or hardware configuration: AWS\r\n- OS (e.g. from /etc/os-release): \r\n```\r\nNAME=\"Container Linux by CoreOS\"\r\nID=coreos\r\nVERSION=1632.3.0\r\nVERSION_ID=1632.3.0\r\nBUILD_ID=2018-02-14-0338\r\nPRETTY_NAME=\"Container Linux by CoreOS 1632.3.0 (Ladybug)\"\r\nANSI_COLOR=\"38;5;75\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://issues.coreos.com\"\r\nCOREOS_BOARD=\"amd64-usr\"\r\n```\r\n- Kernel (e.g. `uname -a`):\r\n```\r\nLinux XXX 4.14.19-coreos #1 SMP Wed Feb 14 03:18:05 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux\r\n```\r\n- Install tools:\r\n- Others:\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/368557230","html_url":"https://github.com/kubernetes/kubernetes/issues/60298#issuecomment-368557230","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60298","id":368557230,"user":{"login":"msau42","id":24448061,"avatar_url":"https://avatars3.githubusercontent.com/u/24448061?v=4","gravatar_id":"","url":"https://api.github.com/users/msau42","html_url":"https://github.com/msau42","followers_url":"https://api.github.com/users/msau42/followers","following_url":"https://api.github.com/users/msau42/following{/other_user}","gists_url":"https://api.github.com/users/msau42/gists{/gist_id}","starred_url":"https://api.github.com/users/msau42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msau42/subscriptions","organizations_url":"https://api.github.com/users/msau42/orgs","repos_url":"https://api.github.com/users/msau42/repos","events_url":"https://api.github.com/users/msau42/events{/privacy}","received_events_url":"https://api.github.com/users/msau42/received_events","type":"User","site_admin":false},"created_at":"2018-02-26T16:18:22Z","updated_at":"2018-02-26T16:18:22Z","author_association":"MEMBER","body":"Hm if PV got deleted then there must be some logs in apiserver that showed who deleted it.  How long ago did it happen? Is it possible it was too long ago and apiserver logs got rotated out?"}}
{"actor":{"display_login":"msau42"}}