{"action":"closed","number":668,"pull_request":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668","id":95841146,"html_url":"https://github.com/PaddlePaddle/Paddle/pull/668","diff_url":"https://github.com/PaddlePaddle/Paddle/pull/668.diff","patch_url":"https://github.com/PaddlePaddle/Paddle/pull/668.patch","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/668","number":668,"state":"closed","locked":false,"title":"Accelerating image processing for CNN","user":{"login":"qingqing01","id":7845005,"avatar_url":"https://avatars.githubusercontent.com/u/7845005?v=3","gravatar_id":"","url":"https://api.github.com/users/qingqing01","html_url":"https://github.com/qingqing01","followers_url":"https://api.github.com/users/qingqing01/followers","following_url":"https://api.github.com/users/qingqing01/following{/other_user}","gists_url":"https://api.github.com/users/qingqing01/gists{/gist_id}","starred_url":"https://api.github.com/users/qingqing01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qingqing01/subscriptions","organizations_url":"https://api.github.com/users/qingqing01/orgs","repos_url":"https://api.github.com/users/qingqing01/repos","events_url":"https://api.github.com/users/qingqing01/events{/privacy}","received_events_url":"https://api.github.com/users/qingqing01/received_events","type":"User","site_admin":false},"body":"The image processing is a key step for CNN. It's slower when reading image list directly by PyDataProvider. And this leads to a lower speedup on Multi-GPUs, especially on 8 GPUs or 16 GPUs.\r\n\r\nThis PR has two features:\r\n\r\n1.  A C++ accelerated library that can be called by Python:  OpenCV + Multi-threads\r\n2.  A Python module with python-multiprocessing and Python-OpenCV\r\n\r\nOne can select one of them to use.\r\n\r\nThe performance on 4 GPUs( K40 ) with input size 224 * 224 *3 is as follows. \r\n实验条件：\r\n- 4 Tesla K40m\r\n- 总batch size: 192\r\n- 下面时间为20个mini-Batch的时间\r\n\r\n1.  PIL直接读取图片list:   64.8s\r\n2.  Pickle对数据打Batch再读取：46.8s\r\n3.  数据打Batch + Python多进程 + Python-OpenCV:  29.7s\r\n4.  数据打Batch + C++多线程 + OpenCV: 26.7s\r\n\r\n下一个PR增加使用文档。\r\n\r\n#457 \r\n\r\n\r\n\r\n","created_at":"2016-11-30T07:34:04Z","updated_at":"2016-12-08T10:09:13Z","closed_at":"2016-12-08T10:09:13Z","merged_at":"2016-12-08T10:09:13Z","merge_commit_sha":"2039070e9cb4dadf12332d16f74cf349fd980d73","assignee":{"login":"reyoung","id":728699,"avatar_url":"https://avatars.githubusercontent.com/u/728699?v=3","gravatar_id":"","url":"https://api.github.com/users/reyoung","html_url":"https://github.com/reyoung","followers_url":"https://api.github.com/users/reyoung/followers","following_url":"https://api.github.com/users/reyoung/following{/other_user}","gists_url":"https://api.github.com/users/reyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/reyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reyoung/subscriptions","organizations_url":"https://api.github.com/users/reyoung/orgs","repos_url":"https://api.github.com/users/reyoung/repos","events_url":"https://api.github.com/users/reyoung/events{/privacy}","received_events_url":"https://api.github.com/users/reyoung/received_events","type":"User","site_admin":false},"assignees":[{"login":"reyoung","id":728699,"avatar_url":"https://avatars.githubusercontent.com/u/728699?v=3","gravatar_id":"","url":"https://api.github.com/users/reyoung","html_url":"https://github.com/reyoung","followers_url":"https://api.github.com/users/reyoung/followers","following_url":"https://api.github.com/users/reyoung/following{/other_user}","gists_url":"https://api.github.com/users/reyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/reyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reyoung/subscriptions","organizations_url":"https://api.github.com/users/reyoung/orgs","repos_url":"https://api.github.com/users/reyoung/repos","events_url":"https://api.github.com/users/reyoung/events{/privacy}","received_events_url":"https://api.github.com/users/reyoung/received_events","type":"User","site_admin":false},{"login":"gangliao","id":3071933,"avatar_url":"https://avatars.githubusercontent.com/u/3071933?v=3","gravatar_id":"","url":"https://api.github.com/users/gangliao","html_url":"https://github.com/gangliao","followers_url":"https://api.github.com/users/gangliao/followers","following_url":"https://api.github.com/users/gangliao/following{/other_user}","gists_url":"https://api.github.com/users/gangliao/gists{/gist_id}","starred_url":"https://api.github.com/users/gangliao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gangliao/subscriptions","organizations_url":"https://api.github.com/users/gangliao/orgs","repos_url":"https://api.github.com/users/gangliao/repos","events_url":"https://api.github.com/users/gangliao/events{/privacy}","received_events_url":"https://api.github.com/users/gangliao/received_events","type":"User","site_admin":false},{"login":"luotao1","id":6836917,"avatar_url":"https://avatars.githubusercontent.com/u/6836917?v=3","gravatar_id":"","url":"https://api.github.com/users/luotao1","html_url":"https://github.com/luotao1","followers_url":"https://api.github.com/users/luotao1/followers","following_url":"https://api.github.com/users/luotao1/following{/other_user}","gists_url":"https://api.github.com/users/luotao1/gists{/gist_id}","starred_url":"https://api.github.com/users/luotao1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luotao1/subscriptions","organizations_url":"https://api.github.com/users/luotao1/orgs","repos_url":"https://api.github.com/users/luotao1/repos","events_url":"https://api.github.com/users/luotao1/events{/privacy}","received_events_url":"https://api.github.com/users/luotao1/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668/commits","review_comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668/comments","review_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/668/comments","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/978d6e8460ed463e7b05b5a4031cb79c700a86f2","head":{"label":"qingqing01:acc_image_proc","ref":"acc_image_proc","sha":"978d6e8460ed463e7b05b5a4031cb79c700a86f2","user":{"login":"qingqing01","id":7845005,"avatar_url":"https://avatars.githubusercontent.com/u/7845005?v=3","gravatar_id":"","url":"https://api.github.com/users/qingqing01","html_url":"https://github.com/qingqing01","followers_url":"https://api.github.com/users/qingqing01/followers","following_url":"https://api.github.com/users/qingqing01/following{/other_user}","gists_url":"https://api.github.com/users/qingqing01/gists{/gist_id}","starred_url":"https://api.github.com/users/qingqing01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qingqing01/subscriptions","organizations_url":"https://api.github.com/users/qingqing01/orgs","repos_url":"https://api.github.com/users/qingqing01/repos","events_url":"https://api.github.com/users/qingqing01/events{/privacy}","received_events_url":"https://api.github.com/users/qingqing01/received_events","type":"User","site_admin":false},"repo":{"id":66909112,"name":"Paddle","full_name":"qingqing01/Paddle","owner":{"login":"qingqing01","id":7845005,"avatar_url":"https://avatars.githubusercontent.com/u/7845005?v=3","gravatar_id":"","url":"https://api.github.com/users/qingqing01","html_url":"https://github.com/qingqing01","followers_url":"https://api.github.com/users/qingqing01/followers","following_url":"https://api.github.com/users/qingqing01/following{/other_user}","gists_url":"https://api.github.com/users/qingqing01/gists{/gist_id}","starred_url":"https://api.github.com/users/qingqing01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qingqing01/subscriptions","organizations_url":"https://api.github.com/users/qingqing01/orgs","repos_url":"https://api.github.com/users/qingqing01/repos","events_url":"https://api.github.com/users/qingqing01/events{/privacy}","received_events_url":"https://api.github.com/users/qingqing01/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/qingqing01/Paddle","description":"PArallel Distributed Deep LEarning","fork":true,"url":"https://api.github.com/repos/qingqing01/Paddle","forks_url":"https://api.github.com/repos/qingqing01/Paddle/forks","keys_url":"https://api.github.com/repos/qingqing01/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/qingqing01/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/qingqing01/Paddle/teams","hooks_url":"https://api.github.com/repos/qingqing01/Paddle/hooks","issue_events_url":"https://api.github.com/repos/qingqing01/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/qingqing01/Paddle/events","assignees_url":"https://api.github.com/repos/qingqing01/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/qingqing01/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/qingqing01/Paddle/tags","blobs_url":"https://api.github.com/repos/qingqing01/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/qingqing01/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/qingqing01/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/qingqing01/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/qingqing01/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/qingqing01/Paddle/languages","stargazers_url":"https://api.github.com/repos/qingqing01/Paddle/stargazers","contributors_url":"https://api.github.com/repos/qingqing01/Paddle/contributors","subscribers_url":"https://api.github.com/repos/qingqing01/Paddle/subscribers","subscription_url":"https://api.github.com/repos/qingqing01/Paddle/subscription","commits_url":"https://api.github.com/repos/qingqing01/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/qingqing01/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/qingqing01/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/qingqing01/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/qingqing01/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/qingqing01/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/qingqing01/Paddle/merges","archive_url":"https://api.github.com/repos/qingqing01/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/qingqing01/Paddle/downloads","issues_url":"https://api.github.com/repos/qingqing01/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/qingqing01/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/qingqing01/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/qingqing01/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/qingqing01/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/qingqing01/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/qingqing01/Paddle/deployments","created_at":"2016-08-30T05:21:17Z","updated_at":"2016-08-30T05:21:20Z","pushed_at":"2016-12-08T08:42:08Z","git_url":"git://github.com/qingqing01/Paddle.git","ssh_url":"git@github.com:qingqing01/Paddle.git","clone_url":"https://github.com/qingqing01/Paddle.git","svn_url":"https://github.com/qingqing01/Paddle","homepage":null,"size":9127,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"PaddlePaddle:develop","ref":"develop","sha":"5d26716c691c0eb7ce3785e48496bea46a6fc63c","user":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars.githubusercontent.com/u/23534030?v=3","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"repo":{"id":65711522,"name":"Paddle","full_name":"PaddlePaddle/Paddle","owner":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars.githubusercontent.com/u/23534030?v=3","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PaddlePaddle/Paddle","description":"PArallel Distributed Deep LEarning","fork":false,"url":"https://api.github.com/repos/PaddlePaddle/Paddle","forks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/forks","keys_url":"https://api.github.com/repos/PaddlePaddle/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PaddlePaddle/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PaddlePaddle/Paddle/teams","hooks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/hooks","issue_events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/events","assignees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/PaddlePaddle/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/tags","blobs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/PaddlePaddle/Paddle/languages","stargazers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/stargazers","contributors_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contributors","subscribers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscribers","subscription_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscription","commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/PaddlePaddle/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PaddlePaddle/Paddle/merges","archive_url":"https://api.github.com/repos/PaddlePaddle/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PaddlePaddle/Paddle/downloads","issues_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/PaddlePaddle/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/PaddlePaddle/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/PaddlePaddle/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/deployments","created_at":"2016-08-15T06:59:08Z","updated_at":"2016-12-08T06:43:55Z","pushed_at":"2016-12-08T10:09:13Z","git_url":"git://github.com/PaddlePaddle/Paddle.git","ssh_url":"git@github.com:PaddlePaddle/Paddle.git","clone_url":"https://github.com/PaddlePaddle/Paddle.git","svn_url":"https://github.com/PaddlePaddle/Paddle","homepage":"http://www.paddlepaddle.org/","size":22582,"stargazers_count":3935,"watchers_count":3935,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":942,"mirror_url":null,"open_issues_count":150,"forks":942,"open_issues":150,"watchers":3935,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668"},"html":{"href":"https://github.com/PaddlePaddle/Paddle/pull/668"},"issue":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/668"},"comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/668/comments"},"review_comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668/comments"},"review_comment":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/668/commits"},"statuses":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/978d6e8460ed463e7b05b5a4031cb79c700a86f2"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"reyoung","id":728699,"avatar_url":"https://avatars.githubusercontent.com/u/728699?v=3","gravatar_id":"","url":"https://api.github.com/users/reyoung","html_url":"https://github.com/reyoung","followers_url":"https://api.github.com/users/reyoung/followers","following_url":"https://api.github.com/users/reyoung/following{/other_user}","gists_url":"https://api.github.com/users/reyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/reyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reyoung/subscriptions","organizations_url":"https://api.github.com/users/reyoung/orgs","repos_url":"https://api.github.com/users/reyoung/repos","events_url":"https://api.github.com/users/reyoung/events{/privacy}","received_events_url":"https://api.github.com/users/reyoung/received_events","type":"User","site_admin":false},"comments":3,"review_comments":20,"commits":8,"additions":279,"deletions":14,"changed_files":2}}
{"actor":{"display_login":"reyoung"}}