{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30990","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30990/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30990/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30990/events","html_url":"https://github.com/kubernetes/kubernetes/issues/30990","id":172116926,"number":30990,"title":"Pods from ReplicationController are sometimes not removed","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"caesarxuchao","id":2823529,"avatar_url":"https://avatars.githubusercontent.com/u/2823529?v=3","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false},"assignees":[{"login":"caesarxuchao","id":2823529,"avatar_url":"https://avatars.githubusercontent.com/u/2823529?v=3","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.4","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22/labels","id":1777384,"number":22,"title":"v1.4","description":"https://github.com/kubernetes/features/blob/master/release-1.4/Release-1.4.md","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":227,"closed_issues":388,"state":"open","created_at":"2016-05-19T18:50:19Z","updated_at":"2016-08-24T19:18:55Z","due_on":"2016-09-15T00:00:00Z","closed_at":null},"comments":16,"created_at":"2016-08-19T12:05:50Z","updated_at":"2016-08-24T19:18:55Z","closed_at":"2016-08-24T19:18:55Z","body":"I'm currently debugging failures from scalability suite.\r\n\r\nIt basically started failing on this run:\r\nhttp://kubekins.dls.corp.google.com:8080/view/Scalability/job/kubernetes-e2e-gce-scalability/10148/\r\n\r\nThis suite is running WITHOUT garbage collector.\r\n\r\nThere are tens of failures like this one:\r\n```\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:197 deleting rc load-small-rc-46 Expected error: <*errors.errorString | 0xc8275da7c0>: { s: \"error while waiting for pods to become inactive load-small-rc-46: timed out waiting for the condition\", } error while waiting for pods to become inactive load-small-rc-46: timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:345\r\n```\r\n\r\nI'm also able to reproduce this problem from head on my local 100-node cluster.\r\n\r\n\r\nWe did some debugging of it and it seems that because of some reason, pods from some replication controllers are simply not being removed.\r\nWhat happens is that:\r\n1. we call: DeleteRCAndPods https://github.com/kubernetes/kubernetes/blob/master/test/e2e/load.go#L345\r\n2. This should result in:\r\n- scaling RC to 0\r\n- waiting until all pods are gone\r\n- removing replication controller\r\n\r\nHowever, what happens (looking into logs) is that the first two doesn't happen, but replication controller is indeed removed. From RC logs (from some run):\r\n```\r\nI0819 11:12:44.621239       7 replication_controller.go:585] Finished syncing controller \"e2e-tests-load-30-nodepods-1-ikyk3/load-small-rc-226\" (5.195731ms)\r\nI0819 11:17:30.168494       7 replication_controller.go:598] Replication Controller has been deleted e2e-tests-load-30-nodepods-1-ikyk3/load-small-rc-226\r\nI0819 11:17:30.168508       7 replication_controller.go:585] Finished syncing controller \"e2e-tests-load-30-nodepods-1-ikyk3/load-small-rc-226\" (16.194�s)\r\n```\r\n\r\nThe timing when it started happening matches exactly merge of #30670, but I don't know how this can affect it.\r\n\r\n@caesarxuchao @lavalamp @gmarek \r\n\r\n\r\nhttp://kubekins.dls.corp.google.com:8080/view/Scalability/job/kubernetes-e2e-gce-scalability/10148/ is the good run to start debugging with"}}
{"actor":{"display_login":"k8s-merge-robot"}}