{"action":"created","comment":{"url":"https://api.github.com/repos/boto/s3transfer/pulls/comments/66363426","id":66363426,"diff_hunk":"@@ -27,26 +29,40 @@\n class TransferConfig(object):\n     def __init__(self,\n                  multipart_threshold=8 * MB,\n-                 max_concurrency=10,\n                  multipart_chunksize=8 * MB,\n-                 max_queue_size=0,\n+                 max_request_concurrency=10,","path":"s3transfer/manager.py","position":29,"original_position":27,"commit_id":"b674d5dea36c505337a024cf07abf5230afadd41","original_commit_id":"ab279dcc9fa08b6c84585b1bc45ed7e5f33fc202","user":{"login":"jamesls","id":368057,"avatar_url":"https://avatars.githubusercontent.com/u/368057?v=3","gravatar_id":"","url":"https://api.github.com/users/jamesls","html_url":"https://github.com/jamesls","followers_url":"https://api.github.com/users/jamesls/followers","following_url":"https://api.github.com/users/jamesls/following{/other_user}","gists_url":"https://api.github.com/users/jamesls/gists{/gist_id}","starred_url":"https://api.github.com/users/jamesls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamesls/subscriptions","organizations_url":"https://api.github.com/users/jamesls/orgs","repos_url":"https://api.github.com/users/jamesls/repos","events_url":"https://api.github.com/users/jamesls/events{/privacy}","received_events_url":"https://api.github.com/users/jamesls/received_events","type":"User","site_admin":false},"body":"Seems reasonable.\r\n\r\nWe'll probably need to loop back and work out good class names for the config classes to make sure no one confuses the similarly named transfer config classes we have.","created_at":"2016-06-09T00:04:41Z","updated_at":"2016-06-09T00:04:41Z","html_url":"https://github.com/boto/s3transfer/pull/13#discussion_r66363426","pull_request_url":"https://api.github.com/repos/boto/s3transfer/pulls/13","_links":{"self":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/comments/66363426"},"html":{"href":"https://github.com/boto/s3transfer/pull/13#discussion_r66363426"},"pull_request":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/13"}}},"pull_request":{"url":"https://api.github.com/repos/boto/s3transfer/pulls/13","id":71125257,"html_url":"https://github.com/boto/s3transfer/pull/13","diff_url":"https://github.com/boto/s3transfer/pull/13.diff","patch_url":"https://github.com/boto/s3transfer/pull/13.patch","issue_url":"https://api.github.com/repos/boto/s3transfer/issues/13","number":13,"state":"open","locked":false,"title":"Wrap task submission for transfer into tasks","user":{"login":"kyleknap","id":4605355,"avatar_url":"https://avatars.githubusercontent.com/u/4605355?v=3","gravatar_id":"","url":"https://api.github.com/users/kyleknap","html_url":"https://github.com/kyleknap","followers_url":"https://api.github.com/users/kyleknap/followers","following_url":"https://api.github.com/users/kyleknap/following{/other_user}","gists_url":"https://api.github.com/users/kyleknap/gists{/gist_id}","starred_url":"https://api.github.com/users/kyleknap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyleknap/subscriptions","organizations_url":"https://api.github.com/users/kyleknap/orgs","repos_url":"https://api.github.com/users/kyleknap/repos","events_url":"https://api.github.com/users/kyleknap/events{/privacy}","received_events_url":"https://api.github.com/users/kyleknap/received_events","type":"User","site_admin":false},"body":"This is a pretty large refactoring of the architecture, but I think that it is worth it and necessary in the long run as we add features. In short, the change wraps the ``TaskSubmitter`` logic that submits all of the tasks to execute a transfer into a ``SubmissionTask`` itself and these submission tasks are then executed by an executor specific for running submission tasks. This is really helpful for the following reasons:\r\n\r\n* The submission of tasks are no longer handled in the main thread, which makes it quicker to enqueue transfer request and get a transfer future back. For example, in downloads and copies you are no longer doing a ``HeadObject ``in the main thread. This is also going to be needed for streaming uploads (i.e. we do not want to stream the entire file in the main thread).\r\n* Exception handling is more robust such that if there was an error during submission before the final task was submitted, bad things could have happened such as hanging or cleanup tasks not being ran. This is no longer the case.\r\n* ``on_queued`` callbacks happen in the submission thread pool. This allows you to make slower calls in these callbacks without slowing down your speed in adding more transfer requests. This will be critical if you want to try to do complete copies for multipart which could require multiple API requests.\r\n\r\nAs to the changes that were made in the refactoring, they include:\r\n\r\n* Remove the ``TaskSubmitter`` class in place of a ``SubmissionTask`` class. They relatively have the same functionality but the ``SubmissionTask`` subclasses from the ``Task`` class so it can be ran by an executor.\r\n* Added a new executor to complete ``SubmissionTask`` that does not interfere with the main executor.\r\n* Moved the running of failure cleanups and done callbacks into the ``TransferCoordinator``. This was changed because logically it made sense that coordinator runs the callbacks associated to it when it announces done especially since it is no longer the responsibility of a single know task to announce that the transfer is done (This can happen in the submission task if it fails or the designated final task).\r\n\r\nLet me know if you have any questions. I tried my best to add comments and documentation at the trickier parts of the code.\r\n\r\ncc @jamesls @JordonPhillips ","created_at":"2016-05-23T21:40:50Z","updated_at":"2016-06-09T00:04:41Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b53c42b20a75e96a47a0d82d81de4ee918383bcd","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/boto/s3transfer/pulls/13/commits","review_comments_url":"https://api.github.com/repos/boto/s3transfer/pulls/13/comments","review_comment_url":"https://api.github.com/repos/boto/s3transfer/pulls/comments{/number}","comments_url":"https://api.github.com/repos/boto/s3transfer/issues/13/comments","statuses_url":"https://api.github.com/repos/boto/s3transfer/statuses/b674d5dea36c505337a024cf07abf5230afadd41","head":{"label":"kyleknap:threaded-submitter","ref":"threaded-submitter","sha":"b674d5dea36c505337a024cf07abf5230afadd41","user":{"login":"kyleknap","id":4605355,"avatar_url":"https://avatars.githubusercontent.com/u/4605355?v=3","gravatar_id":"","url":"https://api.github.com/users/kyleknap","html_url":"https://github.com/kyleknap","followers_url":"https://api.github.com/users/kyleknap/followers","following_url":"https://api.github.com/users/kyleknap/following{/other_user}","gists_url":"https://api.github.com/users/kyleknap/gists{/gist_id}","starred_url":"https://api.github.com/users/kyleknap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyleknap/subscriptions","organizations_url":"https://api.github.com/users/kyleknap/orgs","repos_url":"https://api.github.com/users/kyleknap/repos","events_url":"https://api.github.com/users/kyleknap/events{/privacy}","received_events_url":"https://api.github.com/users/kyleknap/received_events","type":"User","site_admin":false},"repo":{"id":49536530,"name":"s3transfer","full_name":"kyleknap/s3transfer","owner":{"login":"kyleknap","id":4605355,"avatar_url":"https://avatars.githubusercontent.com/u/4605355?v=3","gravatar_id":"","url":"https://api.github.com/users/kyleknap","html_url":"https://github.com/kyleknap","followers_url":"https://api.github.com/users/kyleknap/followers","following_url":"https://api.github.com/users/kyleknap/following{/other_user}","gists_url":"https://api.github.com/users/kyleknap/gists{/gist_id}","starred_url":"https://api.github.com/users/kyleknap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyleknap/subscriptions","organizations_url":"https://api.github.com/users/kyleknap/orgs","repos_url":"https://api.github.com/users/kyleknap/repos","events_url":"https://api.github.com/users/kyleknap/events{/privacy}","received_events_url":"https://api.github.com/users/kyleknap/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kyleknap/s3transfer","description":"[Experimental] Amazon S3 Transfer Manager for Python","fork":true,"url":"https://api.github.com/repos/kyleknap/s3transfer","forks_url":"https://api.github.com/repos/kyleknap/s3transfer/forks","keys_url":"https://api.github.com/repos/kyleknap/s3transfer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kyleknap/s3transfer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kyleknap/s3transfer/teams","hooks_url":"https://api.github.com/repos/kyleknap/s3transfer/hooks","issue_events_url":"https://api.github.com/repos/kyleknap/s3transfer/issues/events{/number}","events_url":"https://api.github.com/repos/kyleknap/s3transfer/events","assignees_url":"https://api.github.com/repos/kyleknap/s3transfer/assignees{/user}","branches_url":"https://api.github.com/repos/kyleknap/s3transfer/branches{/branch}","tags_url":"https://api.github.com/repos/kyleknap/s3transfer/tags","blobs_url":"https://api.github.com/repos/kyleknap/s3transfer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kyleknap/s3transfer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kyleknap/s3transfer/git/refs{/sha}","trees_url":"https://api.github.com/repos/kyleknap/s3transfer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kyleknap/s3transfer/statuses/{sha}","languages_url":"https://api.github.com/repos/kyleknap/s3transfer/languages","stargazers_url":"https://api.github.com/repos/kyleknap/s3transfer/stargazers","contributors_url":"https://api.github.com/repos/kyleknap/s3transfer/contributors","subscribers_url":"https://api.github.com/repos/kyleknap/s3transfer/subscribers","subscription_url":"https://api.github.com/repos/kyleknap/s3transfer/subscription","commits_url":"https://api.github.com/repos/kyleknap/s3transfer/commits{/sha}","git_commits_url":"https://api.github.com/repos/kyleknap/s3transfer/git/commits{/sha}","comments_url":"https://api.github.com/repos/kyleknap/s3transfer/comments{/number}","issue_comment_url":"https://api.github.com/repos/kyleknap/s3transfer/issues/comments{/number}","contents_url":"https://api.github.com/repos/kyleknap/s3transfer/contents/{+path}","compare_url":"https://api.github.com/repos/kyleknap/s3transfer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kyleknap/s3transfer/merges","archive_url":"https://api.github.com/repos/kyleknap/s3transfer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kyleknap/s3transfer/downloads","issues_url":"https://api.github.com/repos/kyleknap/s3transfer/issues{/number}","pulls_url":"https://api.github.com/repos/kyleknap/s3transfer/pulls{/number}","milestones_url":"https://api.github.com/repos/kyleknap/s3transfer/milestones{/number}","notifications_url":"https://api.github.com/repos/kyleknap/s3transfer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kyleknap/s3transfer/labels{/name}","releases_url":"https://api.github.com/repos/kyleknap/s3transfer/releases{/id}","deployments_url":"https://api.github.com/repos/kyleknap/s3transfer/deployments","created_at":"2016-01-12T23:50:28Z","updated_at":"2016-01-12T23:19:42Z","pushed_at":"2016-06-08T20:24:17Z","git_url":"git://github.com/kyleknap/s3transfer.git","ssh_url":"git@github.com:kyleknap/s3transfer.git","clone_url":"https://github.com/kyleknap/s3transfer.git","svn_url":"https://github.com/kyleknap/s3transfer","homepage":null,"size":169,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"boto:develop","ref":"develop","sha":"34fcb8abfa4b04ed945295b7f9eb37f55722d3f2","user":{"login":"boto","id":327752,"avatar_url":"https://avatars.githubusercontent.com/u/327752?v=3","gravatar_id":"","url":"https://api.github.com/users/boto","html_url":"https://github.com/boto","followers_url":"https://api.github.com/users/boto/followers","following_url":"https://api.github.com/users/boto/following{/other_user}","gists_url":"https://api.github.com/users/boto/gists{/gist_id}","starred_url":"https://api.github.com/users/boto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boto/subscriptions","organizations_url":"https://api.github.com/users/boto/orgs","repos_url":"https://api.github.com/users/boto/repos","events_url":"https://api.github.com/users/boto/events{/privacy}","received_events_url":"https://api.github.com/users/boto/received_events","type":"Organization","site_admin":false},"repo":{"id":49535299,"name":"s3transfer","full_name":"boto/s3transfer","owner":{"login":"boto","id":327752,"avatar_url":"https://avatars.githubusercontent.com/u/327752?v=3","gravatar_id":"","url":"https://api.github.com/users/boto","html_url":"https://github.com/boto","followers_url":"https://api.github.com/users/boto/followers","following_url":"https://api.github.com/users/boto/following{/other_user}","gists_url":"https://api.github.com/users/boto/gists{/gist_id}","starred_url":"https://api.github.com/users/boto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boto/subscriptions","organizations_url":"https://api.github.com/users/boto/orgs","repos_url":"https://api.github.com/users/boto/repos","events_url":"https://api.github.com/users/boto/events{/privacy}","received_events_url":"https://api.github.com/users/boto/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/boto/s3transfer","description":"[Experimental] Amazon S3 Transfer Manager for Python","fork":false,"url":"https://api.github.com/repos/boto/s3transfer","forks_url":"https://api.github.com/repos/boto/s3transfer/forks","keys_url":"https://api.github.com/repos/boto/s3transfer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/boto/s3transfer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/boto/s3transfer/teams","hooks_url":"https://api.github.com/repos/boto/s3transfer/hooks","issue_events_url":"https://api.github.com/repos/boto/s3transfer/issues/events{/number}","events_url":"https://api.github.com/repos/boto/s3transfer/events","assignees_url":"https://api.github.com/repos/boto/s3transfer/assignees{/user}","branches_url":"https://api.github.com/repos/boto/s3transfer/branches{/branch}","tags_url":"https://api.github.com/repos/boto/s3transfer/tags","blobs_url":"https://api.github.com/repos/boto/s3transfer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/boto/s3transfer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/boto/s3transfer/git/refs{/sha}","trees_url":"https://api.github.com/repos/boto/s3transfer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/boto/s3transfer/statuses/{sha}","languages_url":"https://api.github.com/repos/boto/s3transfer/languages","stargazers_url":"https://api.github.com/repos/boto/s3transfer/stargazers","contributors_url":"https://api.github.com/repos/boto/s3transfer/contributors","subscribers_url":"https://api.github.com/repos/boto/s3transfer/subscribers","subscription_url":"https://api.github.com/repos/boto/s3transfer/subscription","commits_url":"https://api.github.com/repos/boto/s3transfer/commits{/sha}","git_commits_url":"https://api.github.com/repos/boto/s3transfer/git/commits{/sha}","comments_url":"https://api.github.com/repos/boto/s3transfer/comments{/number}","issue_comment_url":"https://api.github.com/repos/boto/s3transfer/issues/comments{/number}","contents_url":"https://api.github.com/repos/boto/s3transfer/contents/{+path}","compare_url":"https://api.github.com/repos/boto/s3transfer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/boto/s3transfer/merges","archive_url":"https://api.github.com/repos/boto/s3transfer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/boto/s3transfer/downloads","issues_url":"https://api.github.com/repos/boto/s3transfer/issues{/number}","pulls_url":"https://api.github.com/repos/boto/s3transfer/pulls{/number}","milestones_url":"https://api.github.com/repos/boto/s3transfer/milestones{/number}","notifications_url":"https://api.github.com/repos/boto/s3transfer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/boto/s3transfer/labels{/name}","releases_url":"https://api.github.com/repos/boto/s3transfer/releases{/id}","deployments_url":"https://api.github.com/repos/boto/s3transfer/deployments","created_at":"2016-01-12T23:19:42Z","updated_at":"2016-03-21T09:28:45Z","pushed_at":"2016-06-08T20:24:18Z","git_url":"git://github.com/boto/s3transfer.git","ssh_url":"git@github.com:boto/s3transfer.git","clone_url":"https://github.com/boto/s3transfer.git","svn_url":"https://github.com/boto/s3transfer","homepage":null,"size":139,"stargazers_count":6,"watchers_count":6,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":4,"forks":2,"open_issues":4,"watchers":6,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/13"},"html":{"href":"https://github.com/boto/s3transfer/pull/13"},"issue":{"href":"https://api.github.com/repos/boto/s3transfer/issues/13"},"comments":{"href":"https://api.github.com/repos/boto/s3transfer/issues/13/comments"},"review_comments":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/13/comments"},"review_comment":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/boto/s3transfer/pulls/13/commits"},"statuses":{"href":"https://api.github.com/repos/boto/s3transfer/statuses/b674d5dea36c505337a024cf07abf5230afadd41"}}}}
{"actor":{"display_login":"jamesls"}}