{"action":"created","issue":{"url":"https://api.github.com/repos/boto/boto3/issues/691","repository_url":"https://api.github.com/repos/boto/boto3","labels_url":"https://api.github.com/repos/boto/boto3/issues/691/labels{/name}","comments_url":"https://api.github.com/repos/boto/boto3/issues/691/comments","events_url":"https://api.github.com/repos/boto/boto3/issues/691/events","html_url":"https://github.com/boto/boto3/issues/691","id":161791536,"number":691,"title":"buffer size for multipart s3 downloads","user":{"login":"mheilman","id":181744,"avatar_url":"https://avatars.githubusercontent.com/u/181744?v=3","gravatar_id":"","url":"https://api.github.com/users/mheilman","html_url":"https://github.com/mheilman","followers_url":"https://api.github.com/users/mheilman/followers","following_url":"https://api.github.com/users/mheilman/following{/other_user}","gists_url":"https://api.github.com/users/mheilman/gists{/gist_id}","starred_url":"https://api.github.com/users/mheilman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mheilman/subscriptions","organizations_url":"https://api.github.com/users/mheilman/orgs","repos_url":"https://api.github.com/users/mheilman/repos","events_url":"https://api.github.com/users/mheilman/events{/privacy}","received_events_url":"https://api.github.com/users/mheilman/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/boto/boto3/labels/response-needed","name":"response-needed","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-06-22T21:31:18Z","updated_at":"2016-06-23T01:19:20Z","closed_at":null,"body":"I noticed recently that for a large download, the [awscli](https://github.com/aws/aws-cli) (`aws s3 cp s3://...`) was faster than using `boto3.s3.transfer.MultipartDownloader`.\r\n\r\nAfter running a few tests of downloading an 8GB file, it looks like maybe the size of the I/O buffer [here](https://github.com/boto/boto3/blob/2d371d60b6b08b39fabbd60f1c2f2d1277c81de0/boto3/s3/transfer.py#L524) may have something to do with it.  I don't understand why, but making that buffer size larger (e.g., 256KB or 1024KB instead of the current 16KB) seems to improve download speeds consistently for me.\r\n\r\nPerhaps that buffer size should be increased, or maybe just made configurable?  I don't understand the pros and cons other than that making it larger seems to help for my use case.\r\n\r\nTimes for downloading an 8GB file from S3 to a g2.2xlarge instance (I just changed the number in the line of code mentioned above):\r\n\r\n- 100 seconds with 1024KB buffer\r\n- 106 seconds with 256KB buffer\r\n- 118 seconds with 16KB buffer (current boto3 code)\r\n- 256 seconds with 4KB buffer\r\n\r\nCode for testing:\r\n```\r\nimport time\r\nimport boto3\r\nimport logging\r\nfrom concurrent.futures import ProcessPoolExecutor\r\n\r\nt0 = time.time()\r\n\r\nlogging.basicConfig(level='DEBUG')\r\nlogging.getLogger('botocore').setLevel('INFO')\r\nclient = boto3.client('s3')\r\n\r\nconfig = boto3.s3.transfer.TransferConfig(\r\n    multipart_threshold=64 * 1024 * 1024,\r\n    max_concurrency=10,\r\n    num_download_attempts=10,\r\n    multipart_chunksize=16 * 1024 * 1024,\r\n    max_io_queue=10000\r\n)\r\n\r\nconfig = boto3.s3.transfer.TransferConfig()\r\ntransfer = boto3.s3.transfer.MultipartDownloader(client, config, boto3.s3.transfer.OSUtils())\r\ntransfer.download_file('bucket-name', 'path/to/big/file/foo.npy', 'foo2.npy', 8000000000, {})\r\nprint(\"TIME: {} SECONDS\".format(time.time() - t0))\r\n```\r\n\r\nI previously mentioned this [here](https://github.com/boto/boto3/issues/409#issuecomment-224071473)."},"comment":{"url":"https://api.github.com/repos/boto/boto3/issues/comments/227924144","html_url":"https://github.com/boto/boto3/issues/691#issuecomment-227924144","issue_url":"https://api.github.com/repos/boto/boto3/issues/691","id":227924144,"user":{"login":"mheilman","id":181744,"avatar_url":"https://avatars.githubusercontent.com/u/181744?v=3","gravatar_id":"","url":"https://api.github.com/users/mheilman","html_url":"https://github.com/mheilman","followers_url":"https://api.github.com/users/mheilman/followers","following_url":"https://api.github.com/users/mheilman/following{/other_user}","gists_url":"https://api.github.com/users/mheilman/gists{/gist_id}","starred_url":"https://api.github.com/users/mheilman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mheilman/subscriptions","organizations_url":"https://api.github.com/users/mheilman/orgs","repos_url":"https://api.github.com/users/mheilman/repos","events_url":"https://api.github.com/users/mheilman/events{/privacy}","received_events_url":"https://api.github.com/users/mheilman/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T01:19:20Z","updated_at":"2016-06-23T01:19:20Z","body":"Ah, I forgot to mention how long awscli takes above.  awscli takes about 90 seconds on the same machine for the same file, so it's still a bit faster than with boto3 when i changed the io chunksize to 1024KB, but not too much.\r\n\r\nI'll try fiddling around with the multipart_chunksize and/or max_io next..."}}
{"actor":{"display_login":"mheilman"}}