{"action":"reopened","number":2165,"pull_request":{"url":"https://api.github.com/repos/conan-io/conan/pulls/2165","id":158795321,"html_url":"https://github.com/conan-io/conan/pull/2165","diff_url":"https://github.com/conan-io/conan/pull/2165.diff","patch_url":"https://github.com/conan-io/conan/pull/2165.patch","issue_url":"https://api.github.com/repos/conan-io/conan/issues/2165","number":2165,"state":"open","locked":false,"title":"Add support to user-provided copytree function for performance improvement","user":{"login":"sztomi","id":240594,"avatar_url":"https://avatars1.githubusercontent.com/u/240594?v=4","gravatar_id":"","url":"https://api.github.com/users/sztomi","html_url":"https://github.com/sztomi","followers_url":"https://api.github.com/users/sztomi/followers","following_url":"https://api.github.com/users/sztomi/following{/other_user}","gists_url":"https://api.github.com/users/sztomi/gists{/gist_id}","starred_url":"https://api.github.com/users/sztomi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sztomi/subscriptions","organizations_url":"https://api.github.com/users/sztomi/orgs","repos_url":"https://api.github.com/users/sztomi/repos","events_url":"https://api.github.com/users/sztomi/events{/privacy}","received_events_url":"https://api.github.com/users/sztomi/received_events","type":"User","site_admin":false},"body":"On Windows, shutil.copytree performs rather poorly. In the worst cases (like boost with large amount of small files) copying sources often takes more time than building the library.\r\n\r\nThere are faster options for copying large amount of files, either by implementing a better function in python or simply calling a tool that will use the hardware more effectively, such as robocopy or emcopy. \r\n\r\nRather than supporting an individual tool, I think it's best to allow users to provide a shutil.copytree-like function as a ConanFile member which will be picked up and used automatically. This PR adds this capability by checking if the conanfile has a `copytree` attribute available.\r\n\r\n### Benchmark\r\n\r\nI performed the following test with our boost package:\r\n\r\nFirst, I changed the build method to return immediately, so it only performs copying the sources. I made sure that the sources are downloaded before the benchmark and then used the -k switch. \r\n\r\nIn the first case, I didn't change anything else, so the default shutil.copytree copying was performed.\r\n\r\nIn the second case, I added the following to the conanfile:\r\n\r\n```py\r\n  def copytree(self, src_folder, build_folder, symlinks=True, ignore=None):\r\n    import subprocess\r\n    ret = subprocess.run([\"robocopy\", src_folder, build_folder, \"/MT:64\", \"/e\"], \r\n                         stdout=subprocess.PIPE, stderr=subprocess.PIPE).returncode\r\n    assert ret in (0, 1)\r\n```\r\n\r\n**Results**\r\n\r\n```\r\nC:\\Users\\tamas\\work\\conan\\packages\\boost                                                          \r\nλ  Measure-Command { py ..\\..\\envwrap.py py $MYCONAN create plex/stable -pr ..\\..\\profiles\\plex-windows-x86_64-msvc14-release --build boost -k }\r\n                                                                                                                                                \r\n                                                                                                                                                \r\nDays              : 0                                                                                                                           \r\nHours             : 0                                                                                                                           \r\nMinutes           : 4                                                                                                                           \r\nSeconds           : 22                                                                                                                          \r\nMilliseconds      : 642                                                                                                                         \r\nTicks             : 2626422981                                                                                                                  \r\nTotalDays         : 0,00303984141319444                                                                                                         \r\nTotalHours        : 0,0729561939166667                                                                                                          \r\nTotalMinutes      : 4,377371635                                                                                                                 \r\nTotalSeconds      : 262,6422981                                                                                                                 \r\nTotalMilliseconds : 262642,2981                                                                                                                 \r\n                                                                                                                                                \r\n                                                                                                                                                \r\n                                                                                                                                                \r\nC:\\Users\\tamas\\work\\conan\\packages\\boost                                                               \r\nλ  Measure-Command { py ..\\..\\envwrap.py py $MYCONAN create plex/stable -pr ..\\..\\profiles\\plex-windows-x86_64-msvc14-release --build boost -k }\r\n                                                                                                                                                \r\n                                                                                                                                                \r\nDays              : 0                                                                                                                           \r\nHours             : 0                                                                                                                           \r\nMinutes           : 1                                                                                                                           \r\nSeconds           : 58                                                                                                                          \r\nMilliseconds      : 277                                                                                                                         \r\nTicks             : 1182770323                                                                                                                  \r\nTotalDays         : 0,00136894713310185                                                                                                         \r\nTotalHours        : 0,0328547311944444                                                                                                          \r\nTotalMinutes      : 1,97128387166667                                                                                                            \r\nTotalSeconds      : 118,2770323                                                                                                                 \r\nTotalMilliseconds : 118277,0323                                                                                                                 \r\n```\r\n\r\n**That is a 182% improvement** (which obviously adds up to several minutes with many packages), so we would be very interested in this or something similar in conan. Naturally, the `ignore` function is discarded in this naive implementation of the user provided copytree, but I think it's OK for the user to make that decision.\r\n\r\n/cc @tru ","created_at":"2017-12-17T17:32:37Z","updated_at":"2018-05-19T19:31:50Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9311a733c9909b8d11b263866f2d455a2e988564","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/conan-io/conan/pulls/2165/commits","review_comments_url":"https://api.github.com/repos/conan-io/conan/pulls/2165/comments","review_comment_url":"https://api.github.com/repos/conan-io/conan/pulls/comments{/number}","comments_url":"https://api.github.com/repos/conan-io/conan/issues/2165/comments","statuses_url":"https://api.github.com/repos/conan-io/conan/statuses/8c33de4f9c1d2cf3c5a77861b383ed4ae46b2d68","head":{"label":"sztomi:copy_callback","ref":"copy_callback","sha":"8c33de4f9c1d2cf3c5a77861b383ed4ae46b2d68","user":{"login":"sztomi","id":240594,"avatar_url":"https://avatars1.githubusercontent.com/u/240594?v=4","gravatar_id":"","url":"https://api.github.com/users/sztomi","html_url":"https://github.com/sztomi","followers_url":"https://api.github.com/users/sztomi/followers","following_url":"https://api.github.com/users/sztomi/following{/other_user}","gists_url":"https://api.github.com/users/sztomi/gists{/gist_id}","starred_url":"https://api.github.com/users/sztomi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sztomi/subscriptions","organizations_url":"https://api.github.com/users/sztomi/orgs","repos_url":"https://api.github.com/users/sztomi/repos","events_url":"https://api.github.com/users/sztomi/events{/privacy}","received_events_url":"https://api.github.com/users/sztomi/received_events","type":"User","site_admin":false},"repo":{"id":89164113,"name":"conan","full_name":"sztomi/conan","owner":{"login":"sztomi","id":240594,"avatar_url":"https://avatars1.githubusercontent.com/u/240594?v=4","gravatar_id":"","url":"https://api.github.com/users/sztomi","html_url":"https://github.com/sztomi","followers_url":"https://api.github.com/users/sztomi/followers","following_url":"https://api.github.com/users/sztomi/following{/other_user}","gists_url":"https://api.github.com/users/sztomi/gists{/gist_id}","starred_url":"https://api.github.com/users/sztomi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sztomi/subscriptions","organizations_url":"https://api.github.com/users/sztomi/orgs","repos_url":"https://api.github.com/users/sztomi/repos","events_url":"https://api.github.com/users/sztomi/events{/privacy}","received_events_url":"https://api.github.com/users/sztomi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sztomi/conan","description":"Conan.io - The open-source C/C++ package manager","fork":true,"url":"https://api.github.com/repos/sztomi/conan","forks_url":"https://api.github.com/repos/sztomi/conan/forks","keys_url":"https://api.github.com/repos/sztomi/conan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sztomi/conan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sztomi/conan/teams","hooks_url":"https://api.github.com/repos/sztomi/conan/hooks","issue_events_url":"https://api.github.com/repos/sztomi/conan/issues/events{/number}","events_url":"https://api.github.com/repos/sztomi/conan/events","assignees_url":"https://api.github.com/repos/sztomi/conan/assignees{/user}","branches_url":"https://api.github.com/repos/sztomi/conan/branches{/branch}","tags_url":"https://api.github.com/repos/sztomi/conan/tags","blobs_url":"https://api.github.com/repos/sztomi/conan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sztomi/conan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sztomi/conan/git/refs{/sha}","trees_url":"https://api.github.com/repos/sztomi/conan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sztomi/conan/statuses/{sha}","languages_url":"https://api.github.com/repos/sztomi/conan/languages","stargazers_url":"https://api.github.com/repos/sztomi/conan/stargazers","contributors_url":"https://api.github.com/repos/sztomi/conan/contributors","subscribers_url":"https://api.github.com/repos/sztomi/conan/subscribers","subscription_url":"https://api.github.com/repos/sztomi/conan/subscription","commits_url":"https://api.github.com/repos/sztomi/conan/commits{/sha}","git_commits_url":"https://api.github.com/repos/sztomi/conan/git/commits{/sha}","comments_url":"https://api.github.com/repos/sztomi/conan/comments{/number}","issue_comment_url":"https://api.github.com/repos/sztomi/conan/issues/comments{/number}","contents_url":"https://api.github.com/repos/sztomi/conan/contents/{+path}","compare_url":"https://api.github.com/repos/sztomi/conan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sztomi/conan/merges","archive_url":"https://api.github.com/repos/sztomi/conan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sztomi/conan/downloads","issues_url":"https://api.github.com/repos/sztomi/conan/issues{/number}","pulls_url":"https://api.github.com/repos/sztomi/conan/pulls{/number}","milestones_url":"https://api.github.com/repos/sztomi/conan/milestones{/number}","notifications_url":"https://api.github.com/repos/sztomi/conan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sztomi/conan/labels{/name}","releases_url":"https://api.github.com/repos/sztomi/conan/releases{/id}","deployments_url":"https://api.github.com/repos/sztomi/conan/deployments","created_at":"2017-04-23T18:44:32Z","updated_at":"2017-04-23T18:44:35Z","pushed_at":"2017-12-17T11:43:51Z","git_url":"git://github.com/sztomi/conan.git","ssh_url":"git@github.com:sztomi/conan.git","clone_url":"https://github.com/sztomi/conan.git","svn_url":"https://github.com/sztomi/conan","homepage":"https://conan.io","size":3484,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"conan-io:develop","ref":"develop","sha":"f592f2e5235480f487d73475d8d26c67731044c6","user":{"login":"conan-io","id":15212165,"avatar_url":"https://avatars3.githubusercontent.com/u/15212165?v=4","gravatar_id":"","url":"https://api.github.com/users/conan-io","html_url":"https://github.com/conan-io","followers_url":"https://api.github.com/users/conan-io/followers","following_url":"https://api.github.com/users/conan-io/following{/other_user}","gists_url":"https://api.github.com/users/conan-io/gists{/gist_id}","starred_url":"https://api.github.com/users/conan-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/conan-io/subscriptions","organizations_url":"https://api.github.com/users/conan-io/orgs","repos_url":"https://api.github.com/users/conan-io/repos","events_url":"https://api.github.com/users/conan-io/events{/privacy}","received_events_url":"https://api.github.com/users/conan-io/received_events","type":"Organization","site_admin":false},"repo":{"id":47190624,"name":"conan","full_name":"conan-io/conan","owner":{"login":"conan-io","id":15212165,"avatar_url":"https://avatars3.githubusercontent.com/u/15212165?v=4","gravatar_id":"","url":"https://api.github.com/users/conan-io","html_url":"https://github.com/conan-io","followers_url":"https://api.github.com/users/conan-io/followers","following_url":"https://api.github.com/users/conan-io/following{/other_user}","gists_url":"https://api.github.com/users/conan-io/gists{/gist_id}","starred_url":"https://api.github.com/users/conan-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/conan-io/subscriptions","organizations_url":"https://api.github.com/users/conan-io/orgs","repos_url":"https://api.github.com/users/conan-io/repos","events_url":"https://api.github.com/users/conan-io/events{/privacy}","received_events_url":"https://api.github.com/users/conan-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/conan-io/conan","description":"Conan.io - The open-source C/C++ package manager","fork":false,"url":"https://api.github.com/repos/conan-io/conan","forks_url":"https://api.github.com/repos/conan-io/conan/forks","keys_url":"https://api.github.com/repos/conan-io/conan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/conan-io/conan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/conan-io/conan/teams","hooks_url":"https://api.github.com/repos/conan-io/conan/hooks","issue_events_url":"https://api.github.com/repos/conan-io/conan/issues/events{/number}","events_url":"https://api.github.com/repos/conan-io/conan/events","assignees_url":"https://api.github.com/repos/conan-io/conan/assignees{/user}","branches_url":"https://api.github.com/repos/conan-io/conan/branches{/branch}","tags_url":"https://api.github.com/repos/conan-io/conan/tags","blobs_url":"https://api.github.com/repos/conan-io/conan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/conan-io/conan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/conan-io/conan/git/refs{/sha}","trees_url":"https://api.github.com/repos/conan-io/conan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/conan-io/conan/statuses/{sha}","languages_url":"https://api.github.com/repos/conan-io/conan/languages","stargazers_url":"https://api.github.com/repos/conan-io/conan/stargazers","contributors_url":"https://api.github.com/repos/conan-io/conan/contributors","subscribers_url":"https://api.github.com/repos/conan-io/conan/subscribers","subscription_url":"https://api.github.com/repos/conan-io/conan/subscription","commits_url":"https://api.github.com/repos/conan-io/conan/commits{/sha}","git_commits_url":"https://api.github.com/repos/conan-io/conan/git/commits{/sha}","comments_url":"https://api.github.com/repos/conan-io/conan/comments{/number}","issue_comment_url":"https://api.github.com/repos/conan-io/conan/issues/comments{/number}","contents_url":"https://api.github.com/repos/conan-io/conan/contents/{+path}","compare_url":"https://api.github.com/repos/conan-io/conan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/conan-io/conan/merges","archive_url":"https://api.github.com/repos/conan-io/conan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/conan-io/conan/downloads","issues_url":"https://api.github.com/repos/conan-io/conan/issues{/number}","pulls_url":"https://api.github.com/repos/conan-io/conan/pulls{/number}","milestones_url":"https://api.github.com/repos/conan-io/conan/milestones{/number}","notifications_url":"https://api.github.com/repos/conan-io/conan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/conan-io/conan/labels{/name}","releases_url":"https://api.github.com/repos/conan-io/conan/releases{/id}","deployments_url":"https://api.github.com/repos/conan-io/conan/deployments","created_at":"2015-12-01T13:17:02Z","updated_at":"2018-05-19T01:50:37Z","pushed_at":"2018-05-18T14:15:18Z","git_url":"git://github.com/conan-io/conan.git","ssh_url":"git@github.com:conan-io/conan.git","clone_url":"https://github.com/conan-io/conan.git","svn_url":"https://github.com/conan-io/conan","homepage":"https://conan.io","size":4660,"stargazers_count":1845,"watchers_count":1845,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":227,"mirror_url":null,"archived":false,"open_issues_count":371,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":227,"open_issues":371,"watchers":1845,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/conan-io/conan/pulls/2165"},"html":{"href":"https://github.com/conan-io/conan/pull/2165"},"issue":{"href":"https://api.github.com/repos/conan-io/conan/issues/2165"},"comments":{"href":"https://api.github.com/repos/conan-io/conan/issues/2165/comments"},"review_comments":{"href":"https://api.github.com/repos/conan-io/conan/pulls/2165/comments"},"review_comment":{"href":"https://api.github.com/repos/conan-io/conan/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/conan-io/conan/pulls/2165/commits"},"statuses":{"href":"https://api.github.com/repos/conan-io/conan/statuses/8c33de4f9c1d2cf3c5a77861b383ed4ae46b2d68"}},"author_association":"CONTRIBUTOR","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":9,"review_comments":0,"maintainer_can_modify":true,"commits":1,"additions":31,"deletions":1,"changed_files":2}}
{"actor":{"display_login":"lasote"}}