{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21625","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21625/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21625/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21625/events","html_url":"https://github.com/kubernetes/kubernetes/issues/21625","id":135110443,"number":21625,"title":"The servicecontroller loop is racy","user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-20T18:59:35Z","updated_at":"2016-02-20T18:59:35Z","closed_at":null,"body":"In pkg/controller/service/servicecontroller.go, we start 10 goroutines that process service events: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go#L139-L141\r\n\r\nEach of these workers then locks on a per-service mutex when processing events: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go#L225-L228\r\n\r\nBut I believe there is no guarantee that the order of lock acquisition is the same as the order of events.  Thus I think we might process events out of order when services change rapidly.\r\n\r\nAn alternative design might be to have a per-service goroutine with a per-service event queue.  This would mean that service events would be guaranteed to be processed in order.  It would also make caching of the previous state much simpler.  Finally, when we hit an error like a failure to create a LoadBalancer, we could simply sleep without risking exhausting our \"thread pool\"."}}