{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23104","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23104/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23104/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23104/events","html_url":"https://github.com/kubernetes/kubernetes/issues/23104","id":141506134,"number":23104,"title":"Upgrading a node from kubelet 1.1.3 to 1.2.0 results in containers getting destroyed and recreated","user":{"login":"ghodss","id":5209643,"avatar_url":"https://avatars.githubusercontent.com/u/5209643?v=3","gravatar_id":"","url":"https://api.github.com/users/ghodss","html_url":"https://github.com/ghodss","followers_url":"https://api.github.com/users/ghodss/followers","following_url":"https://api.github.com/users/ghodss/following{/other_user}","gists_url":"https://api.github.com/users/ghodss/gists{/gist_id}","starred_url":"https://api.github.com/users/ghodss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ghodss/subscriptions","organizations_url":"https://api.github.com/users/ghodss/orgs","repos_url":"https://api.github.com/users/ghodss/repos","events_url":"https://api.github.com/users/ghodss/events{/privacy}","received_events_url":"https://api.github.com/users/ghodss/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-17T07:55:04Z","updated_at":"2016-03-17T17:03:50Z","closed_at":null,"body":"We upgrade our nodes by upgrading an RPM that stops the kubelet, updates the binary and starts it again. The behavior I'm seeing (100% consistently and reproducibly) is that when I upgrade the RPM, the new kubelet starts up, and after maybe 3-5 seconds starts killing and restarting the containers on the machine. Usually it does all of them, but sometimes it does a subset.\r\n\r\nYou can see the log of the new 1.2 kubelet starting up, up to the point it starts killing containers at https://gist.github.com/ghodss/c579579e53355aed6508.\r\n\r\nThe pods are not evicted from apiserver, and their age does not change.\r\n\r\nDowngrading to 1.1 has the same effect of killing and recreating the containers. Note that I had the exact same thing happen when we upgraded from 1.0.3 to 1.1.3. Let me know if you need any other details.\r\n\r\n@bgrant0607 @dchen1107 @yujuhong \r\n(marking as P0 per @bgrant0607)"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/197975527","html_url":"https://github.com/kubernetes/kubernetes/issues/23104#issuecomment-197975527","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23104","id":197975527,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2016-03-17T17:03:50Z","updated_at":"2016-03-17T17:03:50Z","body":"Sorry for the breakage introduced by this. I did raise the concern when we first worked on #15089, unfortunately docker doesn't support mutable container label. We decided to move on with it because we thought upgrading kubelet binary without kubectl drain is illegal since 1.1 release, and our e2e test never test against this scenario. We didn't realize the customer build their own live-upgrade tools. "}}