{"action":"created","issue":{"url":"https://api.github.com/repos/rusoto/rusoto/issues/1028","repository_url":"https://api.github.com/repos/rusoto/rusoto","labels_url":"https://api.github.com/repos/rusoto/rusoto/issues/1028/labels{/name}","comments_url":"https://api.github.com/repos/rusoto/rusoto/issues/1028/comments","events_url":"https://api.github.com/repos/rusoto/rusoto/issues/1028/events","html_url":"https://github.com/rusoto/rusoto/issues/1028","id":313403164,"number":1028,"title":"S3 upload_part creates 2 extra copies of payload/body in memory","user":{"login":"jpastuszek","id":20052,"avatar_url":"https://avatars3.githubusercontent.com/u/20052?v=4","gravatar_id":"","url":"https://api.github.com/users/jpastuszek","html_url":"https://github.com/jpastuszek","followers_url":"https://api.github.com/users/jpastuszek/followers","following_url":"https://api.github.com/users/jpastuszek/following{/other_user}","gists_url":"https://api.github.com/users/jpastuszek/gists{/gist_id}","starred_url":"https://api.github.com/users/jpastuszek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpastuszek/subscriptions","organizations_url":"https://api.github.com/users/jpastuszek/orgs","repos_url":"https://api.github.com/users/jpastuszek/repos","events_url":"https://api.github.com/users/jpastuszek/events{/privacy}","received_events_url":"https://api.github.com/users/jpastuszek/received_events","type":"User","site_admin":false},"labels":[{"id":577656758,"url":"https://api.github.com/repos/rusoto/rusoto/labels/Easy","name":"Easy","color":"bfdadc","default":false},{"id":236362902,"url":"https://api.github.com/repos/rusoto/rusoto/labels/bug","name":"bug","color":"fc2929","default":true},{"id":236362905,"url":"https://api.github.com/repos/rusoto/rusoto/labels/help%20wanted","name":"help wanted","color":"159818","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-04-11T16:42:00Z","updated_at":"2018-04-20T14:29:42Z","closed_at":null,"author_association":"NONE","body":"Using `S3Client` when calling `upload_part` it will `.clone()` provided body `Vec<u8>` (since it is behind pointer to `UploadPartRequest`. Looking at memory usage of my client there is another copy done somewhere else.\r\nThis results in very high memory usage. For example using 8 parallel part uploads and uploading 280GB file I need to use 28MB parts (since S3 has limit of 10k parts). This results in 700MB of memory usage since we need 3 copies * 28 MB per part * 8 parallel uploads = 672 MB of `Vec<u8>` containing data.\r\n\r\nIn ideal world rusoto would take the `Vec<u8>` as value and return it back with request response for future reuse. Since current API is taking request via pointer this is not possible. I could not find the source of third copy of the buffer though - perhaps that can be easily avoided."},"comment":{"url":"https://api.github.com/repos/rusoto/rusoto/issues/comments/383114455","html_url":"https://github.com/rusoto/rusoto/issues/1028#issuecomment-383114455","issue_url":"https://api.github.com/repos/rusoto/rusoto/issues/1028","id":383114455,"user":{"login":"srijs","id":2677165,"avatar_url":"https://avatars0.githubusercontent.com/u/2677165?v=4","gravatar_id":"","url":"https://api.github.com/users/srijs","html_url":"https://github.com/srijs","followers_url":"https://api.github.com/users/srijs/followers","following_url":"https://api.github.com/users/srijs/following{/other_user}","gists_url":"https://api.github.com/users/srijs/gists{/gist_id}","starred_url":"https://api.github.com/users/srijs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srijs/subscriptions","organizations_url":"https://api.github.com/users/srijs/orgs","repos_url":"https://api.github.com/users/srijs/repos","events_url":"https://api.github.com/users/srijs/events{/privacy}","received_events_url":"https://api.github.com/users/srijs/received_events","type":"User","site_admin":false},"created_at":"2018-04-20T14:29:42Z","updated_at":"2018-04-20T14:29:42Z","author_association":"MEMBER","body":"This is great feedback, thanks!\r\n\r\nThe memory usage on Windows _sounds_ like a bug, but one that most likely lives upstream, in either `hyper` or `tokio`. It might be worth trying to reproduce the issue with a non-rusoto hyper client, and file a bug in the hyper project to follow this up.\r\n\r\nRegarding allowing for re-use of buffers, I've used crates like [`pool`](https://crates.io/crates/pool) for this in the past. Using vanilla `hyper` this would work, since hyper can be generic over buffer types, as long as they implement `AsRef<u8>`. With rusoto, it's a bit more tricky since we've currenly hard-coded the buffer type to `Vec<u8>`, and making it generic will have some pretty big knock-on effects as far as I can tell. We can explore this further if you want, and maybe we can integrate closer with something like `pool` to make this happen.\r\n\r\nLastly, thanks for pointing out the oversight with `content_length`. There was a small bug in the signing logic that prevented me from doing this earlier, but now I'm working on fixing it in #1043."}}
{"actor":{"display_login":"srijs"}}