{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/465","repository_url":"https://api.github.com/repos/coreos/coreos-kubernetes","labels_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/465/labels{/name}","comments_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/465/comments","events_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/465/events","html_url":"https://github.com/coreos/coreos-kubernetes/pull/465","id":153165679,"number":465,"title":"kube-aws: Drain nodes before shutting them down","user":{"login":"mumoshu","id":22009,"avatar_url":"https://avatars.githubusercontent.com/u/22009?v=3","gravatar_id":"","url":"https://api.github.com/users/mumoshu","html_url":"https://github.com/mumoshu","followers_url":"https://api.github.com/users/mumoshu/followers","following_url":"https://api.github.com/users/mumoshu/following{/other_user}","gists_url":"https://api.github.com/users/mumoshu/gists{/gist_id}","starred_url":"https://api.github.com/users/mumoshu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mumoshu/subscriptions","organizations_url":"https://api.github.com/users/mumoshu/orgs","repos_url":"https://api.github.com/users/mumoshu/repos","events_url":"https://api.github.com/users/mumoshu/events{/privacy}","received_events_url":"https://api.github.com/users/mumoshu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/labels/dependency/kubernetes","name":"dependency/kubernetes","color":"fbfa04"},{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/labels/kind/enhancement","name":"kind/enhancement","color":"c7def8"},{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/labels/platform/AWS","name":"platform/AWS","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-05-05T04:24:52Z","updated_at":"2016-05-14T14:47:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/pulls/465","html_url":"https://github.com/coreos/coreos-kubernetes/pull/465","diff_url":"https://github.com/coreos/coreos-kubernetes/pull/465.diff","patch_url":"https://github.com/coreos/coreos-kubernetes/pull/465.patch"},"body":"Drain nodes before shutting them down to give running pods time to gracefully stop.\r\n\r\nThis change basically achieves it by running `docker run IMAGE kubectl drain THE_NODE --force` on the to-be-shut-down node before the kubelet gets SIGTERM in a CoreOS' shutdown process.\r\n\r\nWithout this change, kubelets getting SIGTERM without a prior `drain` results in unfunctional(actually not schedulable) pods to have statuses `Ready`.\r\n\r\nWith this change, when an ASG's desired cap decreased, a node's status changes over time as follows:\r\n\r\nOn desired cap change:\r\n=> STATUS=Ready\r\n\r\nOn shut-down started:\r\n=> STATUS=Ready,SchedulingDisabled (<- Pods are stopped and status is changed by `kubectl drain`)\r\n\r\nOn shut-down finished:\r\n=> Status=NotReady,SchedulingDisabled (<- It's `NotReady` but it won't result in a down-time because we already stopped both further scheduling and pods)\r\n\r\nAfter a minute:\r\n=> The node disappears from the output of `kubectl get nodes`\r\n\r\nNote that:\r\n\r\n* This applies to manual shutdowns(via running `sudo systemctl shutdown` for example) and automated shutdowns(triggered by AWS AutoScaling when nodes get rotated out of a group)\r\n* We currently depend on the community docker image `mumoshu/kubectl` because `kubectl` included in the official `coreos/hyperkube` image doesn't work due to the issue https://github.com/kubernetes/kubernetes/issues/24088 in Kubernetes. Once the issue is fixed and the CoreOS team published the new hyperkube image with the updated Kubernetes, we can remove that dependency.\r\n* I consider this an experimental feature. So you shouldn't expect configuration API regarding this stable. It may change in the future.\r\n\r\npartially fix #340 "},"comment":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/comments/219224176","html_url":"https://github.com/coreos/coreos-kubernetes/pull/465#issuecomment-219224176","issue_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/465","id":219224176,"user":{"login":"philips","id":33544,"avatar_url":"https://avatars.githubusercontent.com/u/33544?v=3","gravatar_id":"","url":"https://api.github.com/users/philips","html_url":"https://github.com/philips","followers_url":"https://api.github.com/users/philips/followers","following_url":"https://api.github.com/users/philips/following{/other_user}","gists_url":"https://api.github.com/users/philips/gists{/gist_id}","starred_url":"https://api.github.com/users/philips/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philips/subscriptions","organizations_url":"https://api.github.com/users/philips/orgs","repos_url":"https://api.github.com/users/philips/repos","events_url":"https://api.github.com/users/philips/events{/privacy}","received_events_url":"https://api.github.com/users/philips/received_events","type":"User","site_admin":false},"created_at":"2016-05-14T14:47:24Z","updated_at":"2016-05-14T14:47:24Z","body":"There is also this issue which takes a different approach for CoreOS autoupdates. They are somewhat orthogonal but highly related: https://github.com/coreos/bugs/issues/1274"}}