{"action":"created","comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments/181908483","pull_request_review_id":112616459,"id":181908483,"diff_hunk":"@@ -0,0 +1,58 @@\n+# Design Doc: Asynchronous Update With Distributed Training\n+\n+## Background\n+\n+For the typical synchronous distributed training, some significant steps are as follows:\n+\n+1. A Trainer will compute the gradients and SEND them to the Parameter Server(PServer) nodes.\n+1. After the PServer node received gradients came from all the Trainers, It will aggregate the\n+gradient variables for the same parameter into one gradient variable and then apply the aggregated\n+gradient to the respective parameter, finally using an optimize algorithms(SGD, Monument...)\n+to update the parameters.\n+1. The Trainer would wait for the PServers finished the optimize stage, and GET the parameters from PServer,\n+so all the Trainers would get the same parameters.\n+\n+In the synchronously distributed training, there should be a `Barrier` to synchronise the\n+parameters after the optimizing stage. The performance of a distributed training job would\n+depend on the slowest node if there were hundreds or thousands of training nodes in a\n+Job, the performance of synchronously distributed training might be very poor because of\n+the slow node. So this design doc would introduce an approach to implement\n+*asynchronously* distributed training in PaddlePaddle Fluid.\n+\n+## Design\n+\n+<img src=\"./src/async_update.png\" width=\"600\"/>\n+\n+As the figure above, we describe a global view of asynchronously update process and use\n+the parameter `w1` as an example to introduce the steps:\n+1. For each gradient variables, they may distribute on different GPU card and aggregate\n+them while they are all calculated.\n+1. Split the gradient variable into multiple blocks according to the number of PServer\n+instances and then sent them.\n+1. PServer would run an `Optimize Block` using a specified optimize algorithm to update\n+the specified parameter.\n+1. The trainer will fetch the parameter before running forward Op depends on the specified\n+parameter.\n+1. Broadcast the received variable into multiple GPU cards and continue to run the next\n+mini-batch.\n+\n+### Trainer\n+\n+- For the multiple devices distributed training, we need to aggregate the gradient\n+variables which placed on different devices firstly and then schedule a `SendVars` Operator to\n+send the gradient variables to the multiple PServer instances.\n+- Schedule `FetchVars` operator to fetch the latest parameter from PServer before running\n+the forward ops.\n+- There could be a large number of gradient variables to be sent, so we need to use another\n+thread pool(IO Threadpool) which a number of the schedulable threads is larger than the\n+computing thread pool to avoid competitive the thread resources with computing.","path":"doc/fluid/design/dist_train/async_update.md","position":48,"original_position":48,"commit_id":"49e885b6eaf969c07413889f12f12241fefbe4f6","original_commit_id":"49e885b6eaf969c07413889f12f12241fefbe4f6","user":{"login":"putcn","id":2062926,"avatar_url":"https://avatars1.githubusercontent.com/u/2062926?v=4","gravatar_id":"","url":"https://api.github.com/users/putcn","html_url":"https://github.com/putcn","followers_url":"https://api.github.com/users/putcn/followers","following_url":"https://api.github.com/users/putcn/following{/other_user}","gists_url":"https://api.github.com/users/putcn/gists{/gist_id}","starred_url":"https://api.github.com/users/putcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/putcn/subscriptions","organizations_url":"https://api.github.com/users/putcn/orgs","repos_url":"https://api.github.com/users/putcn/repos","events_url":"https://api.github.com/users/putcn/events{/privacy}","received_events_url":"https://api.github.com/users/putcn/received_events","type":"User","site_admin":false},"body":"competitive -> competing","created_at":"2018-04-16T22:51:39Z","updated_at":"2018-04-16T22:51:40Z","html_url":"https://github.com/PaddlePaddle/Paddle/pull/9932#discussion_r181908483","pull_request_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments/181908483"},"html":{"href":"https://github.com/PaddlePaddle/Paddle/pull/9932#discussion_r181908483"},"pull_request":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932"}}},"pull_request":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932","id":181748151,"html_url":"https://github.com/PaddlePaddle/Paddle/pull/9932","diff_url":"https://github.com/PaddlePaddle/Paddle/pull/9932.diff","patch_url":"https://github.com/PaddlePaddle/Paddle/pull/9932.patch","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932","number":9932,"state":"open","locked":false,"title":"Add async update design doc","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"body":"Fixed #9925 \r\nTasks: #9941","created_at":"2018-04-16T04:01:18Z","updated_at":"2018-04-16T22:51:40Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4b6cad248ee9ae9f73c62154e6ab02bfa6ca1a76","assignee":null,"assignees":[],"requested_reviewers":[{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/commits","review_comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/comments","review_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932/comments","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/49e885b6eaf969c07413889f12f12241fefbe4f6","head":{"label":"Yancey1989:async_update_design","ref":"async_update_design","sha":"49e885b6eaf969c07413889f12f12241fefbe4f6","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"repo":{"id":83747272,"name":"Paddle","full_name":"Yancey1989/Paddle","owner":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/Yancey1989/Paddle","description":"PArallel Distributed Deep LEarning","fork":true,"url":"https://api.github.com/repos/Yancey1989/Paddle","forks_url":"https://api.github.com/repos/Yancey1989/Paddle/forks","keys_url":"https://api.github.com/repos/Yancey1989/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Yancey1989/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Yancey1989/Paddle/teams","hooks_url":"https://api.github.com/repos/Yancey1989/Paddle/hooks","issue_events_url":"https://api.github.com/repos/Yancey1989/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/Yancey1989/Paddle/events","assignees_url":"https://api.github.com/repos/Yancey1989/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/Yancey1989/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/Yancey1989/Paddle/tags","blobs_url":"https://api.github.com/repos/Yancey1989/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Yancey1989/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Yancey1989/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/Yancey1989/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Yancey1989/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/Yancey1989/Paddle/languages","stargazers_url":"https://api.github.com/repos/Yancey1989/Paddle/stargazers","contributors_url":"https://api.github.com/repos/Yancey1989/Paddle/contributors","subscribers_url":"https://api.github.com/repos/Yancey1989/Paddle/subscribers","subscription_url":"https://api.github.com/repos/Yancey1989/Paddle/subscription","commits_url":"https://api.github.com/repos/Yancey1989/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/Yancey1989/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/Yancey1989/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/Yancey1989/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/Yancey1989/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/Yancey1989/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Yancey1989/Paddle/merges","archive_url":"https://api.github.com/repos/Yancey1989/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Yancey1989/Paddle/downloads","issues_url":"https://api.github.com/repos/Yancey1989/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/Yancey1989/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/Yancey1989/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/Yancey1989/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Yancey1989/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/Yancey1989/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/Yancey1989/Paddle/deployments","created_at":"2017-03-03T02:20:12Z","updated_at":"2017-03-03T02:20:16Z","pushed_at":"2018-04-16T11:43:53Z","git_url":"git://github.com/Yancey1989/Paddle.git","ssh_url":"git@github.com:Yancey1989/Paddle.git","clone_url":"https://github.com/Yancey1989/Paddle.git","svn_url":"https://github.com/Yancey1989/Paddle","homepage":"http://www.paddlepaddle.org/","size":58889,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"PaddlePaddle:develop","ref":"develop","sha":"494c262a26a1ff29143491fa60fd6ba546d3bebf","user":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars3.githubusercontent.com/u/23534030?v=4","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"repo":{"id":65711522,"name":"Paddle","full_name":"PaddlePaddle/Paddle","owner":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars3.githubusercontent.com/u/23534030?v=4","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PaddlePaddle/Paddle","description":"PArallel Distributed Deep LEarning","fork":false,"url":"https://api.github.com/repos/PaddlePaddle/Paddle","forks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/forks","keys_url":"https://api.github.com/repos/PaddlePaddle/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PaddlePaddle/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PaddlePaddle/Paddle/teams","hooks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/hooks","issue_events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/events","assignees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/PaddlePaddle/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/tags","blobs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/PaddlePaddle/Paddle/languages","stargazers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/stargazers","contributors_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contributors","subscribers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscribers","subscription_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscription","commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/PaddlePaddle/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PaddlePaddle/Paddle/merges","archive_url":"https://api.github.com/repos/PaddlePaddle/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PaddlePaddle/Paddle/downloads","issues_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/PaddlePaddle/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/PaddlePaddle/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/PaddlePaddle/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/deployments","created_at":"2016-08-15T06:59:08Z","updated_at":"2018-04-16T17:50:43Z","pushed_at":"2018-04-16T21:42:52Z","git_url":"git://github.com/PaddlePaddle/Paddle.git","ssh_url":"git@github.com:PaddlePaddle/Paddle.git","clone_url":"https://github.com/PaddlePaddle/Paddle.git","svn_url":"https://github.com/PaddlePaddle/Paddle","homepage":"http://www.paddlepaddle.org/","size":99598,"stargazers_count":6722,"watchers_count":6722,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":1811,"mirror_url":null,"archived":false,"open_issues_count":1278,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":1811,"open_issues":1278,"watchers":6722,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932"},"html":{"href":"https://github.com/PaddlePaddle/Paddle/pull/9932"},"issue":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932"},"comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932/comments"},"review_comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/comments"},"review_comment":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/commits"},"statuses":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/49e885b6eaf969c07413889f12f12241fefbe4f6"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"putcn"}}