{"action":"closed","issue":{"url":"https://api.github.com/repos/boto/botocore/issues/1561","repository_url":"https://api.github.com/repos/boto/botocore","labels_url":"https://api.github.com/repos/boto/botocore/issues/1561/labels{/name}","comments_url":"https://api.github.com/repos/boto/botocore/issues/1561/comments","events_url":"https://api.github.com/repos/boto/botocore/issues/1561/events","html_url":"https://github.com/boto/botocore/issues/1561","id":361488016,"node_id":"MDU6SXNzdWUzNjE0ODgwMTY=","number":1561,"title":"Very slow (>60s) API calls with large params","user":{"login":"medmunds","id":639984,"node_id":"MDQ6VXNlcjYzOTk4NA==","avatar_url":"https://avatars0.githubusercontent.com/u/639984?v=4","gravatar_id":"","url":"https://api.github.com/users/medmunds","html_url":"https://github.com/medmunds","followers_url":"https://api.github.com/users/medmunds/followers","following_url":"https://api.github.com/users/medmunds/following{/other_user}","gists_url":"https://api.github.com/users/medmunds/gists{/gist_id}","starred_url":"https://api.github.com/users/medmunds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/medmunds/subscriptions","organizations_url":"https://api.github.com/users/medmunds/orgs","repos_url":"https://api.github.com/users/medmunds/repos","events_url":"https://api.github.com/users/medmunds/events{/privacy}","received_events_url":"https://api.github.com/users/medmunds/received_events","type":"User","site_admin":false},"labels":[{"id":202410410,"node_id":"MDU6TGFiZWwyMDI0MTA0MTA=","url":"https://api.github.com/repos/boto/botocore/labels/investigating","name":"investigating","color":"eb6420","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-18T21:17:56Z","updated_at":"2018-10-12T16:58:09Z","closed_at":"2018-10-12T16:58:09Z","author_association":"NONE","body":"Problem: `boto3.client(\"ses\").send_raw_email(..., RawMessage=b\"message w/ 7MB attachment\")` takes over a minute in AWS Lambda—and only the last few seconds are actually posting to the API endpoint. Most of the time is spent in `urlencode` while _repeatedly_ re-running `AWSPreparedRequest.prepare_body()`.\r\n\r\nThis likely affects any botocore client API using sigv4 auth, though it'll only be noticeable with larger params like SendRawEmail's RawMessage.\r\n\r\n### Issue\r\n\r\nI'm implementing an email forwarder using AWS SES and Lambda, and couldn't figure out why my Lambda function was timing out on messages with attachments. X-Ray showed the problem was in SendRawEmail:\r\n\r\n<img width=\"1095\" alt=\"aws x-ray ses sendrawemail\" src=\"https://user-images.githubusercontent.com/639984/45712410-21d9f280-bb41-11e8-85fb-2dc7cc5d869c.png\">\r\n\r\nA little local profiling identified two problems:\r\n\r\n1. Python's `urlencode` is not particularly fast. Addressing that seems out of scope for boto.\r\n\r\n2. botocore's `AWSPreparedRequest.prepare_body()` is called *9 times* on the same params dict, re-urlencoding those large params every time. This seems excessive.\r\n\r\n[Python 3.6; botocore 1.12.3 local dev <strike>or 1.10.74</strike> and Lambda; 256MB Lambda memory allocation]\r\n\r\n### Fix\r\n\r\nEight of the nine `prepare_body()` calls are due to accessing `request.body` four times in `SigV4Auth`—and the `AWSRequest.body` property implementation unintentionally calls `prepare_body()` twice each time. The ninth is the one you'd expect via `AWSRequest.prepare()` as the request is sent.\r\n\r\nI'm not sure the best way to proceed:\r\n\r\n* No matter what, the [extraneous `prepare_body()` call](https://github.com/boto/botocore/blob/1.12.5/botocore/awsrequest.py#L391) should be removed from `AWSRequest.body`. (The body is already prepared by `AWSPreparedRequest()`'s init, on the preceding line.)\r\n\r\n* A simple improvement for this particular case is to cache `request.body` in auth (e.g., #1478). But `AWSRequest.body` is still an expensive property to compute, which could easily trip up other code the same way. (We could add caching into the body prop implementation itself, but it's not clear to me how to reliably invalidate it.)\r\n\r\n* The more robust fix might be to eliminate `AWSRequest.body` entirely. Code that wants the computed body (like sigv4) would need to explicitly call `AWSRequest.prepare()` and get the body prop from the resulting `AWSPreparedRequest`. That makes it clear there's more going on than a simple property access. (FWIW, the Requests models on which AWSRequest is based do it this way: `requests.Request` does not have a `body` prop; you have to get that from `requests.PreparedRequest`.)\r\n\r\n* Even all that still leaves `urlencode` called twice on the same params: first while [adding auth](https://github.com/boto/botocore/blob/1.12.5/botocore/auth.py#L271-L284) and then again [for the actual send](https://github.com/boto/botocore/blob/1.12.5/botocore/endpoint.py#L117). It would be nice if there were some way to use a single `AWSPreparedRequest` for both, but I gather that doesn't play well with the auth implementation?\r\n\r\n[Corrections: I'd been testing with botocore 1.12.3 in both Lambda and locally, because that was pulled in by aws-xray-sdk. The duplicate `prepare_body` was added sometime after 1.10.74, so it's only run *five* times in the current Lambda environment, but will be run nine times with the latest botocore.]"}}
{"actor":{"display_login":"joguSD"}}