{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21630","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21630/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21630/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21630/events","html_url":"https://github.com/kubernetes/kubernetes/issues/21630","id":135124843,"number":21630,"title":"LoadBalancer creation retry happens every second","user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"quinton-hoole","id":10390785,"avatar_url":"https://avatars.githubusercontent.com/u/10390785?v=3","gravatar_id":"","url":"https://api.github.com/users/quinton-hoole","html_url":"https://github.com/quinton-hoole","followers_url":"https://api.github.com/users/quinton-hoole/followers","following_url":"https://api.github.com/users/quinton-hoole/following{/other_user}","gists_url":"https://api.github.com/users/quinton-hoole/gists{/gist_id}","starred_url":"https://api.github.com/users/quinton-hoole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quinton-hoole/subscriptions","organizations_url":"https://api.github.com/users/quinton-hoole/orgs","repos_url":"https://api.github.com/users/quinton-hoole/repos","events_url":"https://api.github.com/users/quinton-hoole/events{/privacy}","received_events_url":"https://api.github.com/users/quinton-hoole/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/17","html_url":"https://github.com/kubernetes/kubernetes/milestones/next-candidate","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/17/labels","id":1510379,"number":17,"title":"next-candidate","description":"Candidates for the next milestone","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":202,"closed_issues":5,"state":"open","created_at":"2016-01-16T17:58:20Z","updated_at":"2016-02-25T07:11:55Z","due_on":null,"closed_at":null},"comments":14,"created_at":"2016-02-20T21:52:09Z","updated_at":"2016-02-25T22:43:11Z","closed_at":null,"body":"From customer logs, it looks like we retry to create a LoadBalancer after a failure every second.  This causes rate-limiting problems on AWS (and probably will on GCE also?).  We should have a less aggressive retry policy.\r\n\r\nRelated #21625 "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/189016749","html_url":"https://github.com/kubernetes/kubernetes/issues/21630#issuecomment-189016749","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21630","id":189016749,"user":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"created_at":"2016-02-25T22:43:11Z","updated_at":"2016-02-25T22:43:11Z","body":"I redacted all the sensitive stuff.  There is only one LB here, with a name of a6<redacted>\r\n\r\n```\r\nE0220 21:00:26.205601       1 servicecontroller.go:187] Failed to process service delta. Retrying: Failed to create load balancer for service kube-system/kubedash: InvalidConfigurationRequest: ELB cannot be attached to multiple subnets in the same AZ.\r\n    status code: 409, request id: <redacted>\r\nI0220 21:00:26.789398       1 aws_loadbalancer.go:50] Creating load balancer with name: a6<redacted>\r\nE0220 21:00:27.090195       1 servicecontroller.go:187] Failed to process service delta. Retrying: Failed to create load balancer for service kube-system/kubedash: Throttling: Rate exceeded\r\n    status code: 400, request id: <redacted>\r\nI0220 21:00:27.743317       1 aws_loadbalancer.go:50] Creating load balancer with name: a6<redacted>\r\nE0220 21:00:28.113412       1 servicecontroller.go:187] Failed to process service delta. Retrying: Failed to create load balancer for service kube-system/kubedash: Throttling: Rate exceeded\r\n    status code: 400, request id: <redacted>\r\nI0220 21:00:28.697279       1 aws_loadbalancer.go:50] Creating load balancer with name: a6<redacted>\r\nE0220 21:00:29.036503       1 servicecontroller.go:187] Failed to process service delta. Retrying: Failed to create load balancer for service kube-system/kubedash: Throttling: Rate exceeded\r\n```\r\n\r\nI agree that it is every 5 seconds in the code though.\r\n\r\nI think what is happening is we have multiple changes here all being retried: #21952\r\n\r\nWith a relatively small fix proposed in #21982\r\n\r\n"}}