{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60292","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60292/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60292/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60292/events","html_url":"https://github.com/kubernetes/kubernetes/pull/60292","id":299651189,"node_id":"MDExOlB1bGxSZXF1ZXN0MTcwOTcwMTAw","number":60292,"title":"Route controller may trigger cloudprovider api throttle when adding route.","user":{"login":"aoxn","id":5567266,"node_id":"MDQ6VXNlcjU1NjcyNjY=","avatar_url":"https://avatars2.githubusercontent.com/u/5567266?v=4","gravatar_id":"","url":"https://api.github.com/users/aoxn","html_url":"https://github.com/aoxn","followers_url":"https://api.github.com/users/aoxn/followers","following_url":"https://api.github.com/users/aoxn/following{/other_user}","gists_url":"https://api.github.com/users/aoxn/gists{/gist_id}","starred_url":"https://api.github.com/users/aoxn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aoxn/subscriptions","organizations_url":"https://api.github.com/users/aoxn/orgs","repos_url":"https://api.github.com/users/aoxn/repos","events_url":"https://api.github.com/users/aoxn/events{/privacy}","received_events_url":"https://api.github.com/users/aoxn/received_events","type":"User","site_admin":false},"labels":[{"id":477397086,"node_id":"MDU6TGFiZWw0NzczOTcwODY=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":680152113,"node_id":"MDU6TGFiZWw2ODAxNTIxMTM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/do-not-merge/release-note-label-needed","name":"do-not-merge/release-note-label-needed","color":"e11d21","default":false},{"id":778118404,"node_id":"MDU6TGFiZWw3NzgxMTg0MDQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false},{"id":240247879,"node_id":"MDU6TGFiZWwyNDAyNDc4Nzk=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/needs-rebase","name":"needs-rebase","color":"BDBDBD","default":false},{"id":114528223,"node_id":"MDU6TGFiZWwxMTQ1MjgyMjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":116712108,"node_id":"MDU6TGFiZWwxMTY3MTIxMDg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false},{"id":253450895,"node_id":"MDU6TGFiZWwyNTM0NTA4OTU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/S","name":"size/S","color":"77bb00","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2018-02-23T09:40:09Z","updated_at":"2018-07-26T05:31:24Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/60292","html_url":"https://github.com/kubernetes/kubernetes/pull/60292","diff_url":"https://github.com/kubernetes/kubernetes/pull/60292.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/60292.patch"},"body":"Route controller may trigger cloudprovider api throttle when adding route in a noone sleep retry loop. Fix it with exponential backoff retry mode\r\n\r\nSigned-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>\r\n\r\n\r\n\r\n**What this PR does / why we need it**:\r\n\r\n/bug\r\n\r\n**Problem description**\r\n\r\nRoute controller concurrently add route entries using cloudprovider interface with rate limiter.  But it does not sleep in the retry loop. This would trigger out cloudprovider api throttle as it is shown below. \r\n\r\n![image](https://user-images.githubusercontent.com/5567266/36587286-aab5ccca-18bf-11e8-8f70-61d6fe4950cd.png)\r\n```\r\nERROR: logging before flag.Parse: I0209 08:33:58.598316       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze8ts4wal6b3ed90c49 172.60.0.0/24 with hint f1931d87-de6b-11e7-a994-00163e32ec29, throttled 735ns\r\nERROR: logging before flag.Parse: I0209 08:33:58.598336       1 route_controller.go:171] Creating route for node cn-beijing.i-2zebb6e6i2m6zvwc6up6 172.60.3.0/24 with hint 39d1a141-de6e-11e7-8982-00163e321bbb, throttled 521ns\r\nERROR: logging before flag.Parse: I0209 08:33:58.598242       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze1f5i6xucoz94hw3yw 172.60.9.0/24 with hint 9b5ac321-06f7-11e8-8969-00163e321bbb, throttled 963ns\r\nERROR: logging before flag.Parse: I0209 08:33:58.598491       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze7y6caxxnm6z4uo1w8 172.60.2.0/24 with hint 6c20a656-de6c-11e7-be12-00163e321bbb, throttled 539ns\r\nERROR: logging before flag.Parse: I0209 08:33:58.598525       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze8p3a0dhya59c7pmig 172.60.1.0/24 with hint 305f5df2-de6c-11e7-aa85-00163e32ec2c, throttled 482ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.924765       1 route_controller.go:187] Could not create route 9236c287-06f7-11e8-8969-00163e321bbb 172.60.7.0/24 for node cn-beijing.i-2ze1f5i6xucoz94hw3yx after 326.483113ms: Aliyun API Error: RequestId: 77AF6D79-DCD9-415C-B2A0-FDB79C44CF21 Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.924790       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze1f5i6xucoz94hw3yx 172.60.7.0/24 with hint 9236c287-06f7-11e8-8969-00163e321bbb, throttled 848ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.924818       1 route_controller.go:187] Could not create route 305f5df2-de6c-11e7-aa85-00163e32ec2c 172.60.1.0/24 for node cn-beijing.i-2ze8p3a0dhya59c7pmig after 326.277237ms: Aliyun API Error: RequestId: BA17457C-43DF-4D05-B66F-020B8BBCEFE9 Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.924836       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze8p3a0dhya59c7pmig 172.60.1.0/24 with hint 305f5df2-de6c-11e7-aa85-00163e32ec2c, throttled 615ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.926639       1 route_controller.go:187] Could not create route 9b5ac321-06f7-11e8-8969-00163e321bbb 172.60.9.0/24 for node cn-beijing.i-2ze1f5i6xucoz94hw3yw after 328.390643ms: Aliyun API Error: RequestId: 3DE07583-63D3-4E93-9BFE-4695596D06FC Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.926672       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze1f5i6xucoz94hw3yw 172.60.9.0/24 with hint 9b5ac321-06f7-11e8-8969-00163e321bbb, throttled 723ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.929134       1 route_controller.go:187] Could not create route 3c55fe12-de6e-11e7-8982-00163e321bbb 172.60.4.0/24 for node cn-beijing.i-2zebb6e6i2m6zvwc6up7 after 330.89936ms: Aliyun API Error: RequestId: 2BE0CE25-2DB3-4333-96DC-70EFCE51DD48 Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.929152       1 route_controller.go:171] Creating route for node cn-beijing.i-2zebb6e6i2m6zvwc6up7 172.60.4.0/24 with hint 3c55fe12-de6e-11e7-8982-00163e321bbb, throttled 596ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.932355       1 route_controller.go:187] Could not create route 39d1a141-de6e-11e7-8982-00163e321bbb 172.60.3.0/24 for node cn-beijing.i-2zebb6e6i2m6zvwc6up6 after 334.01246ms: Aliyun API Error: RequestId: F536D17A-6E45-4ED4-80A4-D779BAFD3DBF Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.932379       1 route_controller.go:171] Creating route for node cn-beijing.i-2zebb6e6i2m6zvwc6up6 172.60.3.0/24 with hint 39d1a141-de6e-11e7-8982-00163e321bbb, throttled 660ns\r\nERROR: logging before flag.Parse: E0209 08:33:58.936614       1 route_controller.go:187] Could not create route 96451a3d-06f7-11e8-8969-00163e321bbb 172.60.8.0/24 for node cn-beijing.i-2ze1f5i6xucoz94hw3yy after 338.319472ms: Aliyun API Error: RequestId: EDF58E53-210B-4130-948B-15D8A00EE501 Status Code: 400 Code: Throttling Message: Request was denied due to request throttling.\r\nERROR: logging before flag.Parse: I0209 08:33:58.936639       1 route_controller.go:171] Creating route for node cn-beijing.i-2ze1f5i6xucoz94hw3yy 172.60.8.0/24 with hint 96451a3d-06f7-11e8-8969-00163e321bbb, throttled 637ns\r\n\r\n```\r\n\r\n**How to fix?**\r\n\r\nWe add an exponential retry backoff to avoid calling cloudprovider too frequently. SEE changes for more details.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/407981127","html_url":"https://github.com/kubernetes/kubernetes/pull/60292#issuecomment-407981127","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60292","id":407981127,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzk4MTEyNw==","user":{"login":"andrewsykim","id":12699319,"node_id":"MDQ6VXNlcjEyNjk5MzE5","avatar_url":"https://avatars0.githubusercontent.com/u/12699319?v=4","gravatar_id":"","url":"https://api.github.com/users/andrewsykim","html_url":"https://github.com/andrewsykim","followers_url":"https://api.github.com/users/andrewsykim/followers","following_url":"https://api.github.com/users/andrewsykim/following{/other_user}","gists_url":"https://api.github.com/users/andrewsykim/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewsykim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewsykim/subscriptions","organizations_url":"https://api.github.com/users/andrewsykim/orgs","repos_url":"https://api.github.com/users/andrewsykim/repos","events_url":"https://api.github.com/users/andrewsykim/events{/privacy}","received_events_url":"https://api.github.com/users/andrewsykim/received_events","type":"User","site_admin":false},"created_at":"2018-07-26T05:31:24Z","updated_at":"2018-07-26T05:31:24Z","author_association":"MEMBER","body":"/close"}}
{"actor":{"display_login":"andrewsykim"}}