{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23034","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23034/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23034/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23034/events","html_url":"https://github.com/kubernetes/kubernetes/issues/23034","id":141208711,"number":23034,"title":"kube-apiserver service endpoints got flushed after restart controller-manager","user":{"login":"xinxiaogang","id":11433689,"avatar_url":"https://avatars.githubusercontent.com/u/11433689?v=3","gravatar_id":"","url":"https://api.github.com/users/xinxiaogang","html_url":"https://github.com/xinxiaogang","followers_url":"https://api.github.com/users/xinxiaogang/followers","following_url":"https://api.github.com/users/xinxiaogang/following{/other_user}","gists_url":"https://api.github.com/users/xinxiaogang/gists{/gist_id}","starred_url":"https://api.github.com/users/xinxiaogang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xinxiaogang/subscriptions","organizations_url":"https://api.github.com/users/xinxiaogang/orgs","repos_url":"https://api.github.com/users/xinxiaogang/repos","events_url":"https://api.github.com/users/xinxiaogang/events{/privacy}","received_events_url":"https://api.github.com/users/xinxiaogang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-16T08:47:06Z","updated_at":"2016-03-16T08:47:06Z","closed_at":null,"body":"This is a race condition issue for controller-manager especially when we use VIP to connect apiserver for both controller-manager and kube-proxy.\r\n1. During controller-manager restart, an endpoints controller loop will started which responsible for service endpoints sync;\r\n2. Two list-watch thread started which list-watch changes for services and pods;\r\n3. The service list-watch will put changed service into a queue compared with its local cache `serviceStore`;\r\n4. The pod list-watch will load pods from apiserver and store them into its local cache `podStore`;\r\n5. A worker will dequeue a service and try to get all its pods from `podStore` and build endpoints based on this.\r\n6. If the worker find current endpoints is different with original one, it will do service endpoints update to apiserver.\r\n\r\nProblem found on step 4:\r\nDuring restart, usually, service list is faster than pod list, so the worker get `kube-apiserver` service from the queue and try to get its pods from `podStore`. While, at that time, pod list-watch is not complete and the worker got empty pods.\r\nThen, endpoints controller update the `kube-apiserver` endpoints to be empty.\r\nThen, kube-proxy got this change and update its forwarding rule. But iptables still exists which dnat VIP to kube-proxy.\r\nAfter above, both controller-manager and kube-proxy could not connect to apiserver anymore.\r\n\r\nAnd there is no way for them to be automatically recovered."}}