{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-sdk-go/issues/2228","repository_url":"https://api.github.com/repos/aws/aws-sdk-go","labels_url":"https://api.github.com/repos/aws/aws-sdk-go/issues/2228/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-sdk-go/issues/2228/comments","events_url":"https://api.github.com/repos/aws/aws-sdk-go/issues/2228/events","html_url":"https://github.com/aws/aws-sdk-go/issues/2228","id":373874857,"node_id":"MDU6SXNzdWUzNzM4NzQ4NTc=","number":2228,"title":"S3 uploader docs question","user":{"login":"iselind","id":21103057,"node_id":"MDQ6VXNlcjIxMTAzMDU3","avatar_url":"https://avatars1.githubusercontent.com/u/21103057?v=4","gravatar_id":"","url":"https://api.github.com/users/iselind","html_url":"https://github.com/iselind","followers_url":"https://api.github.com/users/iselind/followers","following_url":"https://api.github.com/users/iselind/following{/other_user}","gists_url":"https://api.github.com/users/iselind/gists{/gist_id}","starred_url":"https://api.github.com/users/iselind/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iselind/subscriptions","organizations_url":"https://api.github.com/users/iselind/orgs","repos_url":"https://api.github.com/users/iselind/repos","events_url":"https://api.github.com/users/iselind/events{/privacy}","received_events_url":"https://api.github.com/users/iselind/received_events","type":"User","site_admin":false},"labels":[{"id":154423093,"node_id":"MDU6TGFiZWwxNTQ0MjMwOTM=","url":"https://api.github.com/repos/aws/aws-sdk-go/labels/guidance","name":"guidance","color":"8a19e7","default":false},{"id":172085915,"node_id":"MDU6TGFiZWwxNzIwODU5MTU=","url":"https://api.github.com/repos/aws/aws-sdk-go/labels/response-requested","name":"response-requested","color":"666666","default":false}],"state":"open","locked":false,"assignee":{"login":"diehlaws","id":40442261,"node_id":"MDQ6VXNlcjQwNDQyMjYx","avatar_url":"https://avatars0.githubusercontent.com/u/40442261?v=4","gravatar_id":"","url":"https://api.github.com/users/diehlaws","html_url":"https://github.com/diehlaws","followers_url":"https://api.github.com/users/diehlaws/followers","following_url":"https://api.github.com/users/diehlaws/following{/other_user}","gists_url":"https://api.github.com/users/diehlaws/gists{/gist_id}","starred_url":"https://api.github.com/users/diehlaws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diehlaws/subscriptions","organizations_url":"https://api.github.com/users/diehlaws/orgs","repos_url":"https://api.github.com/users/diehlaws/repos","events_url":"https://api.github.com/users/diehlaws/events{/privacy}","received_events_url":"https://api.github.com/users/diehlaws/received_events","type":"User","site_admin":false},"assignees":[{"login":"diehlaws","id":40442261,"node_id":"MDQ6VXNlcjQwNDQyMjYx","avatar_url":"https://avatars0.githubusercontent.com/u/40442261?v=4","gravatar_id":"","url":"https://api.github.com/users/diehlaws","html_url":"https://github.com/diehlaws","followers_url":"https://api.github.com/users/diehlaws/followers","following_url":"https://api.github.com/users/diehlaws/following{/other_user}","gists_url":"https://api.github.com/users/diehlaws/gists{/gist_id}","starred_url":"https://api.github.com/users/diehlaws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diehlaws/subscriptions","organizations_url":"https://api.github.com/users/diehlaws/orgs","repos_url":"https://api.github.com/users/diehlaws/repos","events_url":"https://api.github.com/users/diehlaws/events{/privacy}","received_events_url":"https://api.github.com/users/diehlaws/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2018-10-25T10:17:34Z","updated_at":"2018-11-26T18:01:13Z","closed_at":null,"author_association":"NONE","body":"Please fill out the sections below to help us address your issue.\r\n\r\n### Version of AWS SDK for Go?\r\nLatest\r\n\r\n### Version of Go (`go version`)?\r\ngo version go1.11 linux/amd64\r\n\r\n### What issue did you see?\r\nWhile using the s3 manager to upload a file, it seems the entire file is read into RAM before it's being sent. The docs give no clue how we can limit this behaviour. Our code base runs in a Kubernetes pod with a restricted amount of available memory.\r\n\r\nOur file's sizes can vary from 1 MB to 1 GB. We need a way to control the RAM consumption while uploading these files.\r\n\r\nWe'd require a way to stream the file to the S3 bucket without having to read the entire file into RAM before sending to S3.\r\n### Steps to reproduce\r\nCreate a Kubernetes pod with a RAM limit of say 10MB and try to upload a 100MB file to any AWS bucket. The pod will be killed by the OOMkiller.\r\n\r\nExample code from our code base:\r\n```go\r\n// UploadCoreFile uploads body combined with meta to the appropriate bucket\r\nfunc UploadCoreFile(body io.Reader, meta map[string]string) (string, error) {\r\n    svc := getSession()\r\n\r\n    uploader := s3manager.NewUploaderWithClient(svc)\r\n\r\n    fname := fmt.Sprintf(\"%s-%s-%s-%d.core\", meta[\"SerialNumber\"], meta[\"ProdNbr\"], meta[\"Version\"], time.Now().Unix())\r\n\r\n    // Upload input parameters\r\n    upParams := &s3manager.UploadInput{\r\n        Bucket: aws.String(coreFileBucket),\r\n        Key:    aws.String(fname),\r\n        Body:   body,\r\n    } \r\n\r\n    // Perform an upload.\r\n    resp, err := uploader.Upload(upParams)\r\n\r\n    if err != nil { \r\n        return \"\", err\r\n    } \r\n    return resp.Location, err\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/aws/aws-sdk-go/issues/comments/441736721","html_url":"https://github.com/aws/aws-sdk-go/issues/2228#issuecomment-441736721","issue_url":"https://api.github.com/repos/aws/aws-sdk-go/issues/2228","id":441736721,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTczNjcyMQ==","user":{"login":"jasdel","id":961963,"node_id":"MDQ6VXNlcjk2MTk2Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/961963?v=4","gravatar_id":"","url":"https://api.github.com/users/jasdel","html_url":"https://github.com/jasdel","followers_url":"https://api.github.com/users/jasdel/followers","following_url":"https://api.github.com/users/jasdel/following{/other_user}","gists_url":"https://api.github.com/users/jasdel/gists{/gist_id}","starred_url":"https://api.github.com/users/jasdel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasdel/subscriptions","organizations_url":"https://api.github.com/users/jasdel/orgs","repos_url":"https://api.github.com/users/jasdel/repos","events_url":"https://api.github.com/users/jasdel/events{/privacy}","received_events_url":"https://api.github.com/users/jasdel/received_events","type":"User","site_admin":false},"created_at":"2018-11-26T18:01:13Z","updated_at":"2018-11-26T18:01:13Z","author_association":"MEMBER","body":"@ncw I think for the SDK to support streaming we need to add additional functionality to the SDK's internals to allow an API parameter to be an `io.Reader`. Currently, a significant portion of the SDK's internal behavior expects a `io.ReadSeeker`. The ReadSeeker is used by both the protocol marshaler for serialization, and signer for computing the request signature.\r\n\r\nS3's PutObject and similar methods are special, and support the magic value `UNSIGNED-PAYLOAD` for the request body's digest. We should investigate how viable it is to update the SDK to allow S3 PutObject to take an io.Reader instead of an io.ReadSeeker."}}
{"actor":{"display_login":"jasdel"}}