{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/packer/issues/6330","repository_url":"https://api.github.com/repos/hashicorp/packer","labels_url":"https://api.github.com/repos/hashicorp/packer/issues/6330/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/packer/issues/6330/comments","events_url":"https://api.github.com/repos/hashicorp/packer/issues/6330/events","html_url":"https://github.com/hashicorp/packer/issues/6330","id":328462835,"node_id":"MDU6SXNzdWUzMjg0NjI4MzU=","number":6330,"title":"Amazon Builder not retrying after RequestLimitExceeded - throttling issue","user":{"login":"CarlInglisBJSS","id":22658343,"node_id":"MDQ6VXNlcjIyNjU4MzQz","avatar_url":"https://avatars0.githubusercontent.com/u/22658343?v=4","gravatar_id":"","url":"https://api.github.com/users/CarlInglisBJSS","html_url":"https://github.com/CarlInglisBJSS","followers_url":"https://api.github.com/users/CarlInglisBJSS/followers","following_url":"https://api.github.com/users/CarlInglisBJSS/following{/other_user}","gists_url":"https://api.github.com/users/CarlInglisBJSS/gists{/gist_id}","starred_url":"https://api.github.com/users/CarlInglisBJSS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CarlInglisBJSS/subscriptions","organizations_url":"https://api.github.com/users/CarlInglisBJSS/orgs","repos_url":"https://api.github.com/users/CarlInglisBJSS/repos","events_url":"https://api.github.com/users/CarlInglisBJSS/events{/privacy}","received_events_url":"https://api.github.com/users/CarlInglisBJSS/received_events","type":"User","site_admin":false},"labels":[{"id":33051538,"node_id":"MDU6TGFiZWwzMzA1MTUzOA==","url":"https://api.github.com/repos/hashicorp/packer/labels/bug","name":"bug","color":"f7c6c7","default":true},{"id":126745464,"node_id":"MDU6TGFiZWwxMjY3NDU0NjQ=","url":"https://api.github.com/repos/hashicorp/packer/labels/builder/amazon","name":"builder/amazon","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-06-01T10:20:12Z","updated_at":"2018-06-02T07:35:49Z","closed_at":null,"author_association":"NONE","body":"Packer version: 1.2.3\r\nPlatform: Linux\r\nBuilder: amazon-ebs\r\n\r\n```\r\n08:28:26 2018/06/01 07:28:24 packer: 2018/06/01 07:28:24 Waiting for state to become: available  \r\n08:28:26 2018/06/01 07:28:24 packer: 2018/06/01 07:28:24 Using 30s as polling delay (change with AWS_POLL_DELAY_SECONDS)  \r\n08:28:26 2018/06/01 07:28:24 packer: 2018/06/01 07:28:24 Allowing 300s to complete (change with AWS_TIMEOUT_SECONDS)  \r\n08:29:33 2018/06/01 07:29:30 packer: 2018/06/01 07:29:30 Error on AMIStateRefresh: RequestLimitExceeded: Request limit exceeded.  \r\n08:29:33 2018/06/01 07:29:30 packer: \tstatus code: 503, request id: 2c9c6471-16fc-45c4-9361-d92b6c5a78a1  \r\n```\r\n\r\nFrom my examination of the code (I'm not a Go programmer) it appears that function `WaitForState` is relying on `isTransientNetworkError` to determine retries, and `503` isn't classed as a retryable error.\r\n\r\nI can't tell if the go sdk is retrying, however I suspect not given how quickly the error is being returned.\r\n\r\nI have opened a support case with Amazon to get more information on the specific request and I will post anything significant that I get back from them to this issue.\r\n\r\nWe do have multiple packers running at the time, however the issue I am concerned about is not that, it's that the builder isn't retrying after throttling.\r\n\r\nIt is of course possible that I've misunderstood something in the Go code. If that's the case I'd appreciate being pointed at my misunderstanding. Thanks.\r\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/packer/issues/comments/394066623","html_url":"https://github.com/hashicorp/packer/issues/6330#issuecomment-394066623","issue_url":"https://api.github.com/repos/hashicorp/packer/issues/6330","id":394066623,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDA2NjYyMw==","user":{"login":"mwhooker","id":15618,"node_id":"MDQ6VXNlcjE1NjE4","avatar_url":"https://avatars3.githubusercontent.com/u/15618?v=4","gravatar_id":"","url":"https://api.github.com/users/mwhooker","html_url":"https://github.com/mwhooker","followers_url":"https://api.github.com/users/mwhooker/followers","following_url":"https://api.github.com/users/mwhooker/following{/other_user}","gists_url":"https://api.github.com/users/mwhooker/gists{/gist_id}","starred_url":"https://api.github.com/users/mwhooker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwhooker/subscriptions","organizations_url":"https://api.github.com/users/mwhooker/orgs","repos_url":"https://api.github.com/users/mwhooker/repos","events_url":"https://api.github.com/users/mwhooker/events{/privacy}","received_events_url":"https://api.github.com/users/mwhooker/received_events","type":"User","site_admin":false},"created_at":"2018-06-02T07:35:49Z","updated_at":"2018-06-02T07:35:49Z","author_association":"COLLABORATOR","body":"@CarlInglisBJSS thanks for the report. Hopefully #6177 will solve this, and it would be really useful if you could help us test that when it’s ready. In the mean time, I would suggest increasing the AWS_POLL_DELAY_SECONDS env var as a workaround."}}
{"actor":{"display_login":"mwhooker"}}