{"action":"created","comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments/181658766","pull_request_review_id":112321521,"id":181658766,"diff_hunk":"@@ -0,0 +1,52 @@\n+# Design Doc: Asynchronous Update With Distributed Training\n+\n+## Background\n+\n+For the typical synchronous distributed training, some significant steps are as follows:\n+\n+1. A Trainer will compute the gradients and SEND them to the Parameter\n+Server(PServer) nodes.\n+1. After the PServer node received gradients came from all the Trainers, it would apply the gradient to the respective variables, and using an optimize algorithms(SGD,\n+ Momentment...) to update the parameters.\n+1. The Trainer would wait for the PServers finished the optimize stage, and GET the parameters from PServer, so all the Trainers would get the same parameters.\n+\n+In the synchronously distributed training, there should be a `Barrier` to synchronise the\n+parameters after the optimizing stage. The performance of a distributed training job \n+depends on the lowest node, if there were hundreds or thousand training nodes in a Job,\n+the performance of synchronously distributed training might be very slow because of \n+the slow node. So this design doc would introduce an approach to implement \n+*asynchronously* distributed training in PaddlePaddle Fluid.\n+\n+## Design\n+\n+<img src=\"./src/async_update.png\" width=\"450\"/>\n+\n+As the figure above, we describe a global view of asynchronously update process and use\n+the parameter `w1` as an example to introduce the steps:\n+1. For each gradient variables, they may distribute on different GPU card and aggregate\n+them while they are all calculated.\n+1. Split the gradient variable into multiple blocks according to the number of PServer\n+instances and sent them.\n+1. PServer would run an `Optimize Block` to use a specified optimize algorithm to update\n+the specified parameter, such as `w1`.\n+1. The trainer will fetch the latest parameter after PServer finished the optimize stage.\n+1. Broadcast the received variable into multiple GPU cards and continue to run the next\n+mini-batch.\n+\n+### Trainer\n+\n+- We need a new Operator named `RemoteOptimize` to send gradients to multiple PServer ","path":"doc/fluid/design/dist_train/async_update.md","position":38,"original_position":38,"commit_id":"8d73752c737d29eb00d5b3ea6bcd754c4c7f93c8","original_commit_id":"8d73752c737d29eb00d5b3ea6bcd754c4c7f93c8","user":{"login":"jacquesqiao","id":3048612,"avatar_url":"https://avatars0.githubusercontent.com/u/3048612?v=4","gravatar_id":"","url":"https://api.github.com/users/jacquesqiao","html_url":"https://github.com/jacquesqiao","followers_url":"https://api.github.com/users/jacquesqiao/followers","following_url":"https://api.github.com/users/jacquesqiao/following{/other_user}","gists_url":"https://api.github.com/users/jacquesqiao/gists{/gist_id}","starred_url":"https://api.github.com/users/jacquesqiao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jacquesqiao/subscriptions","organizations_url":"https://api.github.com/users/jacquesqiao/orgs","repos_url":"https://api.github.com/users/jacquesqiao/repos","events_url":"https://api.github.com/users/jacquesqiao/events{/privacy}","received_events_url":"https://api.github.com/users/jacquesqiao/received_events","type":"User","site_admin":false},"body":"Yes, the `Adam project`(https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-chilimbi.pdf) from Microsoft uses multi-threaded model parameter updates without locks to improve their model accuracy. We can implement it and have a test.","created_at":"2018-04-16T08:35:44Z","updated_at":"2018-04-16T08:35:45Z","html_url":"https://github.com/PaddlePaddle/Paddle/pull/9932#discussion_r181658766","pull_request_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments/181658766"},"html":{"href":"https://github.com/PaddlePaddle/Paddle/pull/9932#discussion_r181658766"},"pull_request":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932"}},"in_reply_to_id":181622089},"pull_request":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932","id":181748151,"html_url":"https://github.com/PaddlePaddle/Paddle/pull/9932","diff_url":"https://github.com/PaddlePaddle/Paddle/pull/9932.diff","patch_url":"https://github.com/PaddlePaddle/Paddle/pull/9932.patch","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932","number":9932,"state":"open","locked":false,"title":"Add async update design doc","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"body":"Fixed #9925 \r\nTasks: #9941","created_at":"2018-04-16T04:01:18Z","updated_at":"2018-04-16T08:35:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1f9cc5b91e26ec20de59e622fee29e4bd0f40b71","assignee":null,"assignees":[],"requested_reviewers":[{"login":"helinwang","id":1724178,"avatar_url":"https://avatars3.githubusercontent.com/u/1724178?v=4","gravatar_id":"","url":"https://api.github.com/users/helinwang","html_url":"https://github.com/helinwang","followers_url":"https://api.github.com/users/helinwang/followers","following_url":"https://api.github.com/users/helinwang/following{/other_user}","gists_url":"https://api.github.com/users/helinwang/gists{/gist_id}","starred_url":"https://api.github.com/users/helinwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/helinwang/subscriptions","organizations_url":"https://api.github.com/users/helinwang/orgs","repos_url":"https://api.github.com/users/helinwang/repos","events_url":"https://api.github.com/users/helinwang/events{/privacy}","received_events_url":"https://api.github.com/users/helinwang/received_events","type":"User","site_admin":false},{"login":"putcn","id":2062926,"avatar_url":"https://avatars1.githubusercontent.com/u/2062926?v=4","gravatar_id":"","url":"https://api.github.com/users/putcn","html_url":"https://github.com/putcn","followers_url":"https://api.github.com/users/putcn/followers","following_url":"https://api.github.com/users/putcn/following{/other_user}","gists_url":"https://api.github.com/users/putcn/gists{/gist_id}","starred_url":"https://api.github.com/users/putcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/putcn/subscriptions","organizations_url":"https://api.github.com/users/putcn/orgs","repos_url":"https://api.github.com/users/putcn/repos","events_url":"https://api.github.com/users/putcn/events{/privacy}","received_events_url":"https://api.github.com/users/putcn/received_events","type":"User","site_admin":false},{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/commits","review_comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/comments","review_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932/comments","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/8d73752c737d29eb00d5b3ea6bcd754c4c7f93c8","head":{"label":"Yancey1989:async_update_design","ref":"async_update_design","sha":"8d73752c737d29eb00d5b3ea6bcd754c4c7f93c8","user":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"repo":{"id":83747272,"name":"Paddle","full_name":"Yancey1989/Paddle","owner":{"login":"Yancey1989","id":1426912,"avatar_url":"https://avatars2.githubusercontent.com/u/1426912?v=4","gravatar_id":"","url":"https://api.github.com/users/Yancey1989","html_url":"https://github.com/Yancey1989","followers_url":"https://api.github.com/users/Yancey1989/followers","following_url":"https://api.github.com/users/Yancey1989/following{/other_user}","gists_url":"https://api.github.com/users/Yancey1989/gists{/gist_id}","starred_url":"https://api.github.com/users/Yancey1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yancey1989/subscriptions","organizations_url":"https://api.github.com/users/Yancey1989/orgs","repos_url":"https://api.github.com/users/Yancey1989/repos","events_url":"https://api.github.com/users/Yancey1989/events{/privacy}","received_events_url":"https://api.github.com/users/Yancey1989/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/Yancey1989/Paddle","description":"PArallel Distributed Deep LEarning","fork":true,"url":"https://api.github.com/repos/Yancey1989/Paddle","forks_url":"https://api.github.com/repos/Yancey1989/Paddle/forks","keys_url":"https://api.github.com/repos/Yancey1989/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Yancey1989/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Yancey1989/Paddle/teams","hooks_url":"https://api.github.com/repos/Yancey1989/Paddle/hooks","issue_events_url":"https://api.github.com/repos/Yancey1989/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/Yancey1989/Paddle/events","assignees_url":"https://api.github.com/repos/Yancey1989/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/Yancey1989/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/Yancey1989/Paddle/tags","blobs_url":"https://api.github.com/repos/Yancey1989/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Yancey1989/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Yancey1989/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/Yancey1989/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Yancey1989/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/Yancey1989/Paddle/languages","stargazers_url":"https://api.github.com/repos/Yancey1989/Paddle/stargazers","contributors_url":"https://api.github.com/repos/Yancey1989/Paddle/contributors","subscribers_url":"https://api.github.com/repos/Yancey1989/Paddle/subscribers","subscription_url":"https://api.github.com/repos/Yancey1989/Paddle/subscription","commits_url":"https://api.github.com/repos/Yancey1989/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/Yancey1989/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/Yancey1989/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/Yancey1989/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/Yancey1989/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/Yancey1989/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Yancey1989/Paddle/merges","archive_url":"https://api.github.com/repos/Yancey1989/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Yancey1989/Paddle/downloads","issues_url":"https://api.github.com/repos/Yancey1989/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/Yancey1989/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/Yancey1989/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/Yancey1989/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Yancey1989/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/Yancey1989/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/Yancey1989/Paddle/deployments","created_at":"2017-03-03T02:20:12Z","updated_at":"2017-03-03T02:20:16Z","pushed_at":"2018-04-16T06:05:43Z","git_url":"git://github.com/Yancey1989/Paddle.git","ssh_url":"git@github.com:Yancey1989/Paddle.git","clone_url":"https://github.com/Yancey1989/Paddle.git","svn_url":"https://github.com/Yancey1989/Paddle","homepage":"http://www.paddlepaddle.org/","size":58564,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"PaddlePaddle:develop","ref":"develop","sha":"494c262a26a1ff29143491fa60fd6ba546d3bebf","user":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars3.githubusercontent.com/u/23534030?v=4","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"repo":{"id":65711522,"name":"Paddle","full_name":"PaddlePaddle/Paddle","owner":{"login":"PaddlePaddle","id":23534030,"avatar_url":"https://avatars3.githubusercontent.com/u/23534030?v=4","gravatar_id":"","url":"https://api.github.com/users/PaddlePaddle","html_url":"https://github.com/PaddlePaddle","followers_url":"https://api.github.com/users/PaddlePaddle/followers","following_url":"https://api.github.com/users/PaddlePaddle/following{/other_user}","gists_url":"https://api.github.com/users/PaddlePaddle/gists{/gist_id}","starred_url":"https://api.github.com/users/PaddlePaddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaddlePaddle/subscriptions","organizations_url":"https://api.github.com/users/PaddlePaddle/orgs","repos_url":"https://api.github.com/users/PaddlePaddle/repos","events_url":"https://api.github.com/users/PaddlePaddle/events{/privacy}","received_events_url":"https://api.github.com/users/PaddlePaddle/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PaddlePaddle/Paddle","description":"PArallel Distributed Deep LEarning","fork":false,"url":"https://api.github.com/repos/PaddlePaddle/Paddle","forks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/forks","keys_url":"https://api.github.com/repos/PaddlePaddle/Paddle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PaddlePaddle/Paddle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PaddlePaddle/Paddle/teams","hooks_url":"https://api.github.com/repos/PaddlePaddle/Paddle/hooks","issue_events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/events{/number}","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/events","assignees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/assignees{/user}","branches_url":"https://api.github.com/repos/PaddlePaddle/Paddle/branches{/branch}","tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/tags","blobs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/refs{/sha}","trees_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/{sha}","languages_url":"https://api.github.com/repos/PaddlePaddle/Paddle/languages","stargazers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/stargazers","contributors_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contributors","subscribers_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscribers","subscription_url":"https://api.github.com/repos/PaddlePaddle/Paddle/subscription","commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/commits{/sha}","git_commits_url":"https://api.github.com/repos/PaddlePaddle/Paddle/git/commits{/sha}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/comments{/number}","issue_comment_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments{/number}","contents_url":"https://api.github.com/repos/PaddlePaddle/Paddle/contents/{+path}","compare_url":"https://api.github.com/repos/PaddlePaddle/Paddle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PaddlePaddle/Paddle/merges","archive_url":"https://api.github.com/repos/PaddlePaddle/Paddle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PaddlePaddle/Paddle/downloads","issues_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues{/number}","pulls_url":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls{/number}","milestones_url":"https://api.github.com/repos/PaddlePaddle/Paddle/milestones{/number}","notifications_url":"https://api.github.com/repos/PaddlePaddle/Paddle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels{/name}","releases_url":"https://api.github.com/repos/PaddlePaddle/Paddle/releases{/id}","deployments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/deployments","created_at":"2016-08-15T06:59:08Z","updated_at":"2018-04-16T07:41:25Z","pushed_at":"2018-04-16T08:23:37Z","git_url":"git://github.com/PaddlePaddle/Paddle.git","ssh_url":"git@github.com:PaddlePaddle/Paddle.git","clone_url":"https://github.com/PaddlePaddle/Paddle.git","svn_url":"https://github.com/PaddlePaddle/Paddle","homepage":"http://www.paddlepaddle.org/","size":99559,"stargazers_count":6717,"watchers_count":6717,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":1813,"mirror_url":null,"archived":false,"open_issues_count":1271,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":1813,"open_issues":1271,"watchers":6717,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932"},"html":{"href":"https://github.com/PaddlePaddle/Paddle/pull/9932"},"issue":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932"},"comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/9932/comments"},"review_comments":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/comments"},"review_comment":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/pulls/9932/commits"},"statuses":{"href":"https://api.github.com/repos/PaddlePaddle/Paddle/statuses/8d73752c737d29eb00d5b3ea6bcd754c4c7f93c8"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jacquesqiao"}}