{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71282","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71282/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71282/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71282/events","html_url":"https://github.com/kubernetes/kubernetes/issues/71282","id":382820959,"node_id":"MDU6SXNzdWUzODI4MjA5NTk=","number":71282,"title":"Add a test for \"graceful\" CSI upgrade between 1.12 to 1.13","user":{"login":"saad-ali","id":10052848,"node_id":"MDQ6VXNlcjEwMDUyODQ4","avatar_url":"https://avatars1.githubusercontent.com/u/10052848?v=4","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":169428334,"node_id":"MDU6TGFiZWwxNjk0MjgzMzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"davidz627","id":5459214,"node_id":"MDQ6VXNlcjU0NTkyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5459214?v=4","gravatar_id":"","url":"https://api.github.com/users/davidz627","html_url":"https://github.com/davidz627","followers_url":"https://api.github.com/users/davidz627/followers","following_url":"https://api.github.com/users/davidz627/following{/other_user}","gists_url":"https://api.github.com/users/davidz627/gists{/gist_id}","starred_url":"https://api.github.com/users/davidz627/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidz627/subscriptions","organizations_url":"https://api.github.com/users/davidz627/orgs","repos_url":"https://api.github.com/users/davidz627/repos","events_url":"https://api.github.com/users/davidz627/events{/privacy}","received_events_url":"https://api.github.com/users/davidz627/received_events","type":"User","site_admin":false},"assignees":[{"login":"davidz627","id":5459214,"node_id":"MDQ6VXNlcjU0NTkyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5459214?v=4","gravatar_id":"","url":"https://api.github.com/users/davidz627","html_url":"https://github.com/davidz627","followers_url":"https://api.github.com/users/davidz627/followers","following_url":"https://api.github.com/users/davidz627/following{/other_user}","gists_url":"https://api.github.com/users/davidz627/gists{/gist_id}","starred_url":"https://api.github.com/users/davidz627/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidz627/subscriptions","organizations_url":"https://api.github.com/users/davidz627/orgs","repos_url":"https://api.github.com/users/davidz627/repos","events_url":"https://api.github.com/users/davidz627/events{/privacy}","received_events_url":"https://api.github.com/users/davidz627/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40","html_url":"https://github.com/kubernetes/kubernetes/milestone/40","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40/labels","id":2913713,"node_id":"MDk6TWlsZXN0b25lMjkxMzcxMw==","number":40,"title":"v1.13","description":"","creator":{"login":"thockin","id":5595220,"node_id":"MDQ6VXNlcjU1OTUyMjA=","avatar_url":"https://avatars0.githubusercontent.com/u/5595220?v=4","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":292,"state":"open","created_at":"2017-11-15T06:32:59Z","updated_at":"2018-11-21T01:31:25Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-11-20T19:45:08Z","updated_at":"2018-11-21T01:52:04Z","closed_at":null,"author_association":"MEMBER","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->\r\n\r\n\r\n**What happened**:\r\n\r\nKubernetes v1.13 supports CSI 1.0.\r\nKubernetes v1.12 and 1.11 support CSI 0.3).\r\n\r\nCSI 1.0 is not backwards compatible with CSI 0.3 (see https://github.com/kubernetes/kubernetes/issues/71228#issuecomment-440434852).\r\n\r\nTherefore the upgrade from an earlier version to 1.13 will be disruptive for anyone depending on the CSI driver.\r\n\r\nA vendor may choose to make a version of a CSI driver that supports both versions. But that is unlikely.\r\n\r\nThe disruptive upgrade path is: Remove the old CSI drivers from the cluster before upgrading. Upgrade master and node to k8s v1.13, and wait until master and all nodes are upgraded to 1.13, and then deploy the new CSI 1.0 compatible version of the driver.\r\n\r\nThis approach will be disruptive for the duration of the cluster upgrade as it would mean that during the upgrade no volumes of that type can be provisioned/attached/mounted/etc. After the cluster/driver upgrade things should just start working again with the existing PV/PVCs/objects (unless the driver name changed, or the driver had some other change between 0.3 and 1.0 that prevents it from operating on the old PV/PVC/Storageclass objects).\r\n\r\n**What you expected to happen**:\r\n\r\nA possible path for non-disruptive upgrade:\r\n\r\nBefore upgrade:\r\n\r\n1. Deploy 2 driver daemonsets: one that is 1.12 (CSI 0.3) compatible and one that is 1.13 (CSI 1.0) compatible.\r\n2. Replace the CSI 0.3 compatible driver controller with a CSI 1.0 compatible driver controller.\r\n\r\nAfter the upgrade:\r\n\r\n1. Remove the CSI 0.3 compatible daemonset.\r\n\r\nThis *should* enable non-disruptive upgrades as long as the driver name does not change across upgrades, and other then CSI version the driver remains backwards compatible.\r\n\r\nThis is possible because v1.13 changes the CSI driver registration directory, and a 1.12 kubelet will ignore the CSI 1.0 deployment, and a 1.13 kubelet will ignore the CSI 0.3 deployment.\r\n\r\nAction item here is to write an upgrade test to verify this.\r\n\r\nCC @davidz627 @msau42 @AishSundar @msau42 \r\n\r\n/sig storage\r\n/milestone v1.13\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\n**Anything else we need to know?**:\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n- Cloud provider or hardware configuration:\r\n- OS (e.g. from /etc/os-release):\r\n- Kernel (e.g. `uname -a`):\r\n- Install tools:\r\n- Others:\r\n\r\n<!-- DO NOT EDIT BELOW THIS LINE -->\r\n/kind bug"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/440498872","html_url":"https://github.com/kubernetes/kubernetes/issues/71282#issuecomment-440498872","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/71282","id":440498872,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDQ5ODg3Mg==","user":{"login":"davidz627","id":5459214,"node_id":"MDQ6VXNlcjU0NTkyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5459214?v=4","gravatar_id":"","url":"https://api.github.com/users/davidz627","html_url":"https://github.com/davidz627","followers_url":"https://api.github.com/users/davidz627/followers","following_url":"https://api.github.com/users/davidz627/following{/other_user}","gists_url":"https://api.github.com/users/davidz627/gists{/gist_id}","starred_url":"https://api.github.com/users/davidz627/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidz627/subscriptions","organizations_url":"https://api.github.com/users/davidz627/orgs","repos_url":"https://api.github.com/users/davidz627/repos","events_url":"https://api.github.com/users/davidz627/events{/privacy}","received_events_url":"https://api.github.com/users/davidz627/received_events","type":"User","site_admin":false},"created_at":"2018-11-21T01:52:04Z","updated_at":"2018-11-21T01:52:04Z","author_association":"CONTRIBUTOR","body":"@saad-ali here's my state dump for today. \r\n\r\nAttempted to investigate this today but was unable to upgrade a cluster from `1.12->1.13` not for lack of trying. \r\n\r\nSome learnings and thoughts:\r\n- Deployments of driver pods has to be namespaced or have different deployment names as to not collide\r\n- `registration-dir` must be changed to `DirectoryOrCreate` since the new ` /var/lib/kubelet/plugins_registry/` directory doesn't exist in `1.12`\r\n- Its possible that registration may not work properly after upgrade to `1.13` due to socket already pre-existing in the directory. Not sure if the Plugin Watcher will reconcile or if it just reacts to \"new\" sockets.\r\n  - @RenaudWasTaken might be able to confirm/deny\r\n- Driver name must be stable between versions (for storage class to be stable (for finding the driver to do operations on a pre-existing pod to work))\r\n- Naive method of running double driver will have 2 versions of `external-provisioner` and `external-attacher` running which will race to react to `PVC` and `VolumeAttachment` objects respectively.\r\n  - User would have to modify \"old\" deployment to remove controller components and only deploy \"new\" controller components, provisioning/attaching will be either down or racey in transition state depending on ordering of operations\r\n- We should have docs somewhere for updating clusters or have the scripts be easier to use\r\n\r\nAttempted (and unsuccessful) upgrade methods:\r\n`cluster/gce/upgrade.sh ci/latest`\r\n`kubetest --provider=gce --gcp-project=dyzz-test --test --check-version-skew=false --test_args=\"--ginkgo.focus=\\[Feature:ClusterUpgrade\\]\"`\r\n`kubetest --provider=gce --gcp-project=dyzz-test --test --check-version-skew=false --test_args=\"--ginkgo.focus=\\[Feature:ClusterUpgrade\\]\" --upgrade_args=\"--upgrade-target=ci/latest\"`\r\nnone of which were particularly successful."}}
{"actor":{"display_login":"davidz627"}}