{"action":"created","issue":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/536","repository_url":"https://api.github.com/repos/cloudnativelabs/kube-router","labels_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/536/labels{/name}","comments_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/536/comments","events_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/536/events","html_url":"https://github.com/cloudnativelabs/kube-router/issues/536","id":360036538,"node_id":"MDU6SXNzdWUzNjAwMzY1Mzg=","number":536,"title":"Reduce api calls for disableSourceDestinationCheck","user":{"login":"buddyledungarees","id":3676980,"node_id":"MDQ6VXNlcjM2NzY5ODA=","avatar_url":"https://avatars1.githubusercontent.com/u/3676980?v=4","gravatar_id":"","url":"https://api.github.com/users/buddyledungarees","html_url":"https://github.com/buddyledungarees","followers_url":"https://api.github.com/users/buddyledungarees/followers","following_url":"https://api.github.com/users/buddyledungarees/following{/other_user}","gists_url":"https://api.github.com/users/buddyledungarees/gists{/gist_id}","starred_url":"https://api.github.com/users/buddyledungarees/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buddyledungarees/subscriptions","organizations_url":"https://api.github.com/users/buddyledungarees/orgs","repos_url":"https://api.github.com/users/buddyledungarees/repos","events_url":"https://api.github.com/users/buddyledungarees/events{/privacy}","received_events_url":"https://api.github.com/users/buddyledungarees/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-13T19:36:17Z","updated_at":"2018-09-14T15:54:03Z","closed_at":null,"author_association":"NONE","body":"I have a 100 node cluster and we're seeing the same problem as #313. We're running v0.2.0 of kube-router (latest release as of this issue).\r\n\r\nI'm wondering if we can reduce the number of calls by having the nodes only try to disable the setting for themselves and only on startup? We run kube-router as a daemonset anyway and the volume of calls over a 30 minute window was averaging 2.5 per sec across all kubernetes nodes in one subnet. Granted that doesn't seem like much but AWS doesn't publish their throttling limits and empirically, we're getting throttled:\r\n\r\n```\r\nE0913 17:09:13.559902       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: 504aa627-d3e8-47e5-b4ae-503aa310fb16\r\nE0913 17:09:35.458108       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: 490e1305-c47f-41d5-9ac2-82e670446ae9\r\nE0913 17:10:03.918433       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: de771669-47e2-458b-8636-0095417347a9\r\nE0913 17:10:28.118314       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: 1da40c06-0c3e-4591-8685-484e9b4a6e21\r\nE0913 17:10:49.015117       1 health_controller.go:125] NetworkService Controller heartbeat missed\r\nE0913 17:10:54.008204       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: ce86e736-de49-4a25-8189-1d5d43946e0a\r\nE0913 17:11:18.179957       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: de0551a6-393b-403c-9e0b-c9c52c71d447\r\nE0913 17:11:43.252905       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: 78b47071-7414-4761-a9b0-86036d07ba07\r\nE0913 17:11:54.033868       1 health_controller.go:125] NetworkService Controller heartbeat missed\r\nE0913 17:12:05.077176       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: a9307324-6dd7-40d3-907d-aae7877c260a\r\nE0913 17:12:29.952005       1 aws.go:61] Failed to disable source destination check due to: RequestLimitExceeded: Request limit exceeded.\r\n        status code: 503, request id: 614eddfb-0688-408b-8ab6-19d616851600\r\n```\r\n\r\nWould it make more sense to have each node disable just themselves?"},"comment":{"url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/comments/421402764","html_url":"https://github.com/cloudnativelabs/kube-router/issues/536#issuecomment-421402764","issue_url":"https://api.github.com/repos/cloudnativelabs/kube-router/issues/536","id":421402764,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMTQwMjc2NA==","user":{"login":"murali-reddy","id":2202956,"node_id":"MDQ6VXNlcjIyMDI5NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/2202956?v=4","gravatar_id":"","url":"https://api.github.com/users/murali-reddy","html_url":"https://github.com/murali-reddy","followers_url":"https://api.github.com/users/murali-reddy/followers","following_url":"https://api.github.com/users/murali-reddy/following{/other_user}","gists_url":"https://api.github.com/users/murali-reddy/gists{/gist_id}","starred_url":"https://api.github.com/users/murali-reddy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/murali-reddy/subscriptions","organizations_url":"https://api.github.com/users/murali-reddy/orgs","repos_url":"https://api.github.com/users/murali-reddy/repos","events_url":"https://api.github.com/users/murali-reddy/events{/privacy}","received_events_url":"https://api.github.com/users/murali-reddy/received_events","type":"User","site_admin":false},"created_at":"2018-09-14T15:54:03Z","updated_at":"2018-09-14T15:54:03Z","author_association":"MEMBER","body":"Like you pointed we tried to optimise in #313 but looks like problem still persists.\r\n\r\nSo currently if a node (EC2 instance) does not have IAM to disable source-destination checks it does not try further. In your deployment does each node has IAM to disable src-dst check?\r\n\r\n> Would it make more sense to have each node disable just themselves?\r\n\r\n I am not sure that works for all deployments. For e.g. if i remember correctly in a kops provisioned cluster only master nodes have necessary IAM so the approach in kube-router seemed to have worked reasonably. So some nodes has to the job of disabling src-dst checks.\r\n\r\nBut I understand there should more resilient approach in kube-router. We are open to have any ideas that can work without running into AWS throttling issues."}}
{"actor":{"display_login":"murali-reddy"}}