{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62171","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62171/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62171/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62171/events","html_url":"https://github.com/kubernetes/kubernetes/issues/62171","id":311670282,"node_id":"MDU6SXNzdWUzMTE2NzAyODI=","number":62171,"title":"AWS does not fail when provisioning a volume encrypted with inaccessible KSM key.","user":{"login":"jsafrane","id":1745006,"node_id":"MDQ6VXNlcjE3NDUwMDY=","avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":328487998,"node_id":"MDU6TGFiZWwzMjg0ODc5OTg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/aws","name":"sig/aws","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"jsafrane","id":1745006,"node_id":"MDQ6VXNlcjE3NDUwMDY=","avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false},"assignees":[{"login":"jsafrane","id":1745006,"node_id":"MDQ6VXNlcjE3NDUwMDY=","avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false}],"milestone":null,"comments":11,"created_at":"2018-04-05T15:57:19Z","updated_at":"2018-06-19T12:18:04Z","closed_at":null,"author_association":"MEMBER","body":"This is continuation of #48438 and #48936.\r\n\r\n**What happened**:\r\nWhen StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later.\r\n\r\nRelated code is [here](https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go#L2205) - `checkEncryptionKey` checks only for `NotFoundException`. Other errors are just logged and provisioning continues.\r\n\r\nProblem is that when AWS really can't access the key, it **will** provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist.\r\n\r\n**What you expected to happen**:\r\nKubernetes must not create PV in this case and must report error during provisioning of such volume.\r\n\r\nIdeas:\r\n\r\n* According to [AWS documentation](https://docs.aws.amazon.com/kms/latest/developerguide/services-ebs.html), the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user.\r\n\r\n* Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X?\r\n\r\n* Kubernetes volume plugin for AWS could return both a warning \"I failed to check encryption key, created volume may not work\" that would be propagated to user **and** the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal.\r\n\r\n* Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here.\r\n\r\n/kind bug\r\n/sig aws"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/398378430","html_url":"https://github.com/kubernetes/kubernetes/issues/62171#issuecomment-398378430","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/62171","id":398378430,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODM3ODQzMA==","user":{"login":"jsafrane","id":1745006,"node_id":"MDQ6VXNlcjE3NDUwMDY=","avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false},"created_at":"2018-06-19T12:18:04Z","updated_at":"2018-06-19T12:18:04Z","author_association":"MEMBER","body":"I got an idea, the volume is \"creating\" and then it gets \"available\". I could `DescribeVolumes` in a loop until newly created volume either gets \"Available\" or deleted. This will abuse the API quota (`Describe*` is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?"}}
{"actor":{"display_login":"jsafrane"}}