{"action":"created","issue":{"url":"https://api.github.com/repos/datawire/ambassador/issues/1789","repository_url":"https://api.github.com/repos/datawire/ambassador","labels_url":"https://api.github.com/repos/datawire/ambassador/issues/1789/labels{/name}","comments_url":"https://api.github.com/repos/datawire/ambassador/issues/1789/comments","events_url":"https://api.github.com/repos/datawire/ambassador/issues/1789/events","html_url":"https://github.com/datawire/ambassador/issues/1789","id":489173206,"node_id":"MDU6SXNzdWU0ODkxNzMyMDY=","number":1789,"title":"Crash with endpoint resolver with a lot of endpoints","user":{"login":"concaf","id":9133660,"node_id":"MDQ6VXNlcjkxMzM2NjA=","avatar_url":"https://avatars1.githubusercontent.com/u/9133660?v=4","gravatar_id":"","url":"https://api.github.com/users/concaf","html_url":"https://github.com/concaf","followers_url":"https://api.github.com/users/concaf/followers","following_url":"https://api.github.com/users/concaf/following{/other_user}","gists_url":"https://api.github.com/users/concaf/gists{/gist_id}","starred_url":"https://api.github.com/users/concaf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/concaf/subscriptions","organizations_url":"https://api.github.com/users/concaf/orgs","repos_url":"https://api.github.com/users/concaf/repos","events_url":"https://api.github.com/users/concaf/events{/privacy}","received_events_url":"https://api.github.com/users/concaf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-09-04T13:49:42Z","updated_at":"2019-12-26T12:39:51Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"A user has reported the following on Slack. We need to reproduce this and confirm.\r\n\r\n```\r\nankur Aug 29th at 9:47 AM\r\nWe are running application with 170 micro-services and 280 endpoints. While configuring ambassador with endpoint resolver. It is not able to start. I increased initial delay (for readiness and liveliness) to 600 seconds. Still it shows logs like following and exiting with exit code 137\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=ui-web-api-gateway|*\r\n2019/08/29 04:13:49 aggregator: initialized k8s watch: endpoints|default|metadata.name=ui-web-server|*\r\n2019/08/29 04:13:49 aggregator: initialized k8s watch: endpoints|default|metadata.name=ui-web-service|*\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=unit-test-engine|*\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=view|*\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=web-api|*\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=web-api-orchestrator|*\r\n2019/08/29 04:13:49 aggregator: waiting for k8s watch: endpoints|default|metadata.name=web-api-reporting|*\r\nIs there any way to configure my services with endpoint resolver?\r\n\r\nrdl  6 days ago\r\nWhat version? Could you try it on a cluster with fewer endpoints to see if it's a performance issue or config issue? (e.g., with 2 endpoints or something like that)\r\n\r\nankur  6 days ago\r\nI tried with 100 endpoints and 300 secs intialdelay. It works\r\n\r\nankur  6 days ago\r\nBigger issue for us is:\r\n• With ServiceResolver configuration, load balancing get unbalanced, due to Kubernetes service is L4 load balancer.\r\n• With EndpointResolver doesn't works with 280 Endpoints it doesn't start at all. (with initialDelay of 600 seconds) and we are expecting 400 endpoints in production.\r\nSo question is Ambassador really tested with large cluster?\r\n```"},"comment":{"url":"https://api.github.com/repos/datawire/ambassador/issues/comments/569051863","html_url":"https://github.com/datawire/ambassador/issues/1789#issuecomment-569051863","issue_url":"https://api.github.com/repos/datawire/ambassador/issues/1789","id":569051863,"node_id":"MDEyOklzc3VlQ29tbWVudDU2OTA1MTg2Mw==","user":{"login":"tribonacci","id":24840065,"node_id":"MDQ6VXNlcjI0ODQwMDY1","avatar_url":"https://avatars0.githubusercontent.com/u/24840065?v=4","gravatar_id":"","url":"https://api.github.com/users/tribonacci","html_url":"https://github.com/tribonacci","followers_url":"https://api.github.com/users/tribonacci/followers","following_url":"https://api.github.com/users/tribonacci/following{/other_user}","gists_url":"https://api.github.com/users/tribonacci/gists{/gist_id}","starred_url":"https://api.github.com/users/tribonacci/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tribonacci/subscriptions","organizations_url":"https://api.github.com/users/tribonacci/orgs","repos_url":"https://api.github.com/users/tribonacci/repos","events_url":"https://api.github.com/users/tribonacci/events{/privacy}","received_events_url":"https://api.github.com/users/tribonacci/received_events","type":"User","site_admin":false},"created_at":"2019-12-26T12:39:51Z","updated_at":"2019-12-26T12:39:51Z","author_association":"NONE","body":"Facing same issue here . With 140 services and 300 endpoints ambassador is taking around 280sec to . start."}}
{"actor":{"display_login":"tribonacci"}}