{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32466","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32466/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32466/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32466/events","html_url":"https://github.com/kubernetes/kubernetes/issues/32466","id":176267573,"number":32466,"title":"Make cluster/gce/upgrade.sh and kube-down.sh more helpful when the rolling upgrade fails","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-11T21:52:59Z","updated_at":"2016-09-11T21:52:59Z","closed_at":null,"body":"`cluster/gce/upgrade.sh` uses `gcloud alpha compute rolling-updates` under the covers to do node upgrade. If the MIG rolling update wedges for some reason, we need to give the user enough information to recover (automatically recovering is probably too error-prone). One possibility is that when it starts node upgrade, upgrade.sh should print out the commands for monitoring the status and canceling (canceling requires pausing first, then canceling). Something like\r\n```\r\nYou can monitor the progress of the upgrade using\r\n% gcloud alpha compute rolling-updates describe XXX\r\nand\r\n% gcloud alpha compute rolling-updates list-instance-updates XXX\r\n\r\nTo pause, resume, cancel, or rollback the upgrade, first kill the upgrade.sh script and then use\r\n% gcloud alpha compute rolling-updates <pause/resume/cancel/rollback> XXX\r\n(to cancel, you must pause first and then cancel).\r\n\r\nIf you kill the upgrade, you'll need to remove the old instance template using\r\ngcloud compute instance-templates delete YYY\r\n```\r\nwhere YYY is the old instance template name and XXX is the rolling update ID (we should fill these in for the user). In large clusters there will be more than one MIG, hence more than one rolling-update, so we'd need to list all of the XXX's (i.e. multiple `gcloud alpha compute rolling-updates describe` and `list-instance-updates` and `<pause/resume/cancel/rollback>` lines).\r\n\r\nWe should also have `kube-down.sh` notice if there are any rolling upgrades that are in any state other than CANCELLED, and print a message telling the user to pause and then cancel them if they think they should not be running, using language similar to the above.\r\n\r\ncc/ @gmarek @wojtek-t @grodrigues3 \r\n"}}
{"actor":{"display_login":"davidopp"}}