{"action":"created","issue":{"url":"https://api.github.com/repos/convox/rack/issues/1133","repository_url":"https://api.github.com/repos/convox/rack","labels_url":"https://api.github.com/repos/convox/rack/issues/1133/labels{/name}","comments_url":"https://api.github.com/repos/convox/rack/issues/1133/comments","events_url":"https://api.github.com/repos/convox/rack/issues/1133/events","html_url":"https://github.com/convox/rack/issues/1133","id":173011395,"number":1133,"title":"AWS Throttling Exceptions","user":{"login":"bobzoller","id":16850,"avatar_url":"https://avatars.githubusercontent.com/u/16850?v=3","gravatar_id":"","url":"https://api.github.com/users/bobzoller","html_url":"https://github.com/bobzoller","followers_url":"https://api.github.com/users/bobzoller/followers","following_url":"https://api.github.com/users/bobzoller/following{/other_user}","gists_url":"https://api.github.com/users/bobzoller/gists{/gist_id}","starred_url":"https://api.github.com/users/bobzoller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobzoller/subscriptions","organizations_url":"https://api.github.com/users/bobzoller/orgs","repos_url":"https://api.github.com/users/bobzoller/repos","events_url":"https://api.github.com/users/bobzoller/events{/privacy}","received_events_url":"https://api.github.com/users/bobzoller/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/convox/rack/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-24T17:25:29Z","updated_at":"2016-09-08T18:59:52Z","closed_at":null,"body":"We're just short of being plagued by AWS Throttling issues and need some help.\r\n\r\nIt seems like they occur most frequently during deploys of apps that have many (> 3) processes.  These are coming from CloudFormation DescribeStacks calls being made by the convox builder process.\r\n\r\nBut perhaps more worrisome is I've also seen scheduled jobs fail to start because the ecs-agent itself gets a throttling exception while pulling the container from ECR.  For jobs that only run daily, this means another 24 hours until it retries.  My unconfirmed hunch is this happens most often during timeframes where multiple scheduled jobs are firing.\r\n\r\nI've spent the last few days going around with AWS support in the hopes that I could just get a rate limit increase, but I've just been denied.  Here's the last thing they said on the subject:\r\n\r\n> Hello and welcome back,\r\n> \r\n> Thank you for providing your logs, I have had a look through them and I can see the DescribeStacks API calls being made from your go sdk and also a lambda function. Most of the calls seem to come from the go sdk, for example:\r\n>     {\r\n>       \"eventVersion\": \"1.04\",\r\n>       \"userIdentity\": {\r\n>         \"type\": \"IAMUser\",\r\n>         \"principalId\": \"AIDAIB22BVMEYUM5ZFWEE\",\r\n>         \"arn\": \"arn:aws:iam::XXX:user/convox/convox-KernelUser-1XJXG1XKLXEHY\",\r\n>         \"accountId\": \"XXX\",\r\n>         \"accessKeyId\": \"YYY\",\r\n>         \"userName\": \"convox-KernelUser-1XJXG1XKLXEHY\"\r\n>       },\r\n>       \"eventTime\": \"2016-08-21T02:50:16Z\",\r\n>       \"eventSource\": \"cloudformation.amazonaws.com\",\r\n>       \"eventName\": \"DescribeStacks\",\r\n>       \"awsRegion\": \"us-east-1\",\r\n>       \"sourceIPAddress\": \"52.90.31.20\",\r\n>       \"userAgent\": \"aws-sdk-go/1.1.18 (go1.6.2; linux; amd64)\",\r\n>       \"requestParameters\": {\r\n>         \"stackName\": \"convox-status\"\r\n>       },\r\n>       \"responseElements\": null,\r\n>       \"requestID\": \"fdca5d41-6749-11e6-a775-1bdec881938c\",\r\n>       \"eventID\": \"cafea5f2-ae3a-4722-8372-209909178eef\",\r\n>       \"eventType\": \"AwsApiCall\",\r\n>       \"recipientAccountId\": \"XXX\"\r\n>     },\r\n> \r\n> From looking at those particular logs for just the DescribeStacks API, it would appear that the calls would range between 2-3 API calls a second which would be in range in my opinion. However I cannot see all of the other API calls being made to CloudFormation so it is hard to say exactly what is causing the throttling.\r\n> \r\n> I had a chat with once of our CloudFormation Ops team members and unfortunately with the small amount of stacks you have, the service team will most likely deny an API limit increase. They would consider it if you had 1000's of stacks, but at the moment I would recommend you try the work around's I gave you in my previous correspondence.\r\n> \r\n> These work arounds include using the \"DependsOn\" attribute in your template which would spread out the API calls so that so many calls won't be made at once. The other option was to increase expontential backoff retries or space out the retries so which would significantly help your issue.\r\n\r\nHelp!  Our migration to Convox has ground to a halt until we can work around this.  Happy to provide more logs, info, or help on PRs.\r\n\r\n/cc @nzoschke \r\n"},"comment":{"url":"https://api.github.com/repos/convox/rack/issues/comments/245703815","html_url":"https://github.com/convox/rack/issues/1133#issuecomment-245703815","issue_url":"https://api.github.com/repos/convox/rack/issues/1133","id":245703815,"user":{"login":"bobzoller","id":16850,"avatar_url":"https://avatars.githubusercontent.com/u/16850?v=3","gravatar_id":"","url":"https://api.github.com/users/bobzoller","html_url":"https://github.com/bobzoller","followers_url":"https://api.github.com/users/bobzoller/followers","following_url":"https://api.github.com/users/bobzoller/following{/other_user}","gists_url":"https://api.github.com/users/bobzoller/gists{/gist_id}","starred_url":"https://api.github.com/users/bobzoller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobzoller/subscriptions","organizations_url":"https://api.github.com/users/bobzoller/orgs","repos_url":"https://api.github.com/users/bobzoller/repos","events_url":"https://api.github.com/users/bobzoller/events{/privacy}","received_events_url":"https://api.github.com/users/bobzoller/received_events","type":"User","site_admin":false},"created_at":"2016-09-08T18:59:52Z","updated_at":"2016-09-08T18:59:52Z","body":"12 hrs on `20160827184744` (which includes #1147) and we haven't had a deploy throttled yet.  decent signal but if we make it another 24-48 hrs without that'll be a great signal."}}
{"actor":{"display_login":"bobzoller"}}