{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32323","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32323/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32323/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32323/events","html_url":"https://github.com/kubernetes/kubernetes/issues/32323","id":175881461,"number":32323,"title":"Pods are deleted across node upgrade","user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.4","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22/labels","id":1777384,"number":22,"title":"v1.4","description":"https://github.com/kubernetes/features/blob/master/release-1.4/Release-1.4.md","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":62,"closed_issues":810,"state":"open","created_at":"2016-05-19T18:50:19Z","updated_at":"2016-09-09T17:39:42Z","due_on":"2016-09-20T00:00:00Z","closed_at":null},"comments":16,"created_at":"2016-09-08T22:31:25Z","updated_at":"2016-09-09T18:01:50Z","closed_at":"2016-09-09T08:40:38Z","body":"During node upgrade it looks like sometimes the node can leave the cloudprovider, leading the nodecontroller to think it should delete pods on the node. This is obviously disruptive, but if the pod is not managed by an RC, it can just disappear, which is worse. \r\n\r\nCreate a job\r\n```yaml\r\napiVersion: batch/v1\r\nkind: Job\r\nmetadata:\r\n  name: echo-success\r\nspec:\r\n  template:\r\n    metadata:\r\n      name: echo-success\r\n    spec:\r\n      containers:\r\n      - name: c\r\n        image: busybox\r\n        command: [\"/bin/sh\",  \"-c\", \"echo success\"]\r\n      restartPolicy: Never\r\n  parallelism: 3\r\n  completions: 3\r\n```\r\n\r\nWait for it to succeed and check job:\r\n```\r\n$ k get job \r\nNAME           DESIRED   SUCCESSFUL   AGE\r\necho-success   3         3            31m\r\n\r\n$ k get po --show-all \r\nNAME                        READY     STATUS      RESTARTS   AGE       IP            NODE\r\necho-success-70guk   0/1       Completed   0          13m\r\necho-success-83k2b   0/1       Completed   0          13m\r\necho-success-csjo9   0/1       Completed   0          13m\r\n```\r\n\r\nCheck nodes\r\n```\r\nNAME                           STATUS                     AGE\r\nkubernetes-master              Ready,SchedulingDisabled   30s\r\nkubernetes-minion-group-a9np   Ready                      34s\r\nkubernetes-minion-group-bj5r   Ready                      35s\r\nkubernetes-minion-group-c26b   Ready                      34s\r\n```\r\n\r\nUpgrade nodes\r\n```\r\n$ cluster/gce/upgrade.sh -N v1.4.0-beta.0\r\n```\r\n\r\nCheck nodes\r\n```\r\nNAME                           STATUS                     AGE\r\nkubernetes-master              Ready,SchedulingDisabled   33m\r\nkubernetes-minion-group-a9np   Ready                      11m\r\nkubernetes-minion-group-bj5r   Ready                      9m\r\nkubernetes-minion-group-c26b   Ready                      7m\r\n```\r\n\r\nCheck jobs\r\n```\r\n$ k get job\r\nNAME           DESIRED   SUCCESSFUL   AGE\r\necho-success   3         3            29m\r\n\r\n$ k get po --show-all -o wide\r\nNAME                 READY     STATUS      RESTARTS   AGE       IP           NODE\r\necho-success-83k2b   0/1       Completed   0          29m       10.244.1.4   kubernetes-minion-group-a9np\r\n```\r\n\r\nThis is confusing because the job says 3 but I can only see 1 when I know I saw 3 before upgrade. \r\nWhat's worse, I can't see its logs\r\n```\r\n$ k logs echo-success-83k2b\r\nError response from daemon: No such container: 35935b7065a03f7ac60edf203c1cdc19aaec6696ddbb1036aebf74509457a30d\r\n```\r\n\r\nNode controller has\r\n```\r\nI0908 22:08:28.687091       5 deletion_utils.go:273] Recording Deleting Node kubernetes-minion-group-bj5r because it's not present according to cloud provider event message for node kubernetes-minion-group-bj5r\r\nI0908 22:08:33.689852       5 deletion_utils.go:273] Recording Removing Node kubernetes-minion-group-bj5r from NodeController event message for node kubernetes-minion-group-bj5r\r\nI0908 22:08:33.744494       5 deletion_utils.go:273] Recording Deleting all Pods from Node kubernetes-minion-group-bj5r. event message for node kubernetes-minion-group-bj5r\r\nW0908 22:08:55.886551       5 deletion_utils.go:168] Unable to find Node: kubernetes-minion-group-bj5r, deleting all assigned Pods.\r\n```\r\n\r\n@kubernetes/goog-cluster @kubernetes/goog-gke fyi for slo around node upgrade\r\nI don't know if we want to do anything about this, marking p0 1.4 for triage. "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/245991402","html_url":"https://github.com/kubernetes/kubernetes/issues/32323#issuecomment-245991402","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/32323","id":245991402,"user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"created_at":"2016-09-09T18:01:50Z","updated_at":"2016-09-09T18:01:50Z","body":"It's worse for everyone who changes the hostnames of their nodes (with or without upgrade, this will take 5m and is treated as a netsplit). I think there're a few cases here:     \r\n\r\n1. User deletes node\r\n2. Cloudprovider randomly changes nodename\r\n3. Upgrade ends up deleting node    \r\n    a. It comes back with same nodename\r\n    b. it comes back with different nodename\r\n\r\nUx is worse in 3.b, because it'll take 5m to get working pods post upgrade. It is basically treated as a netsplit, because hostname changed. \r\n\r\n(Ux is also worse in 1, but do users actually care about 5m? they deleted the node, if they want to evict asap, they can delete pods too right)\r\n\r\n@justinsb how does AWS handle upgrade? "}}
{"actor":{"display_login":"bprashanth"}}