{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-ml/issues/171","repository_url":"https://api.github.com/repos/dask/dask-ml","labels_url":"https://api.github.com/repos/dask/dask-ml/issues/171/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-ml/issues/171/comments","events_url":"https://api.github.com/repos/dask/dask-ml/issues/171/events","html_url":"https://github.com/dask/dask-ml/issues/171","id":324897219,"node_id":"MDU6SXNzdWUzMjQ4OTcyMTk=","number":171,"title":"Parameter Server","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":39,"created_at":"2018-05-21T12:24:42Z","updated_at":"2018-08-08T15:39:56Z","closed_at":null,"author_association":"MEMBER","body":"I'm restarting the discussion on parameter servers from https://github.com/dask/dask-glm/issues/57\r\n\r\nWhen doing distributed training it is often useful to rapidly update and share parameters between distributed workers.  The frequency with which these updates can be shared seems to strongly affect the rate of convergence.  The common practice today seems to be to have dedicated nodes within the network that serve as parameter servers, accepting and aggregating updates from workers, and periodically publishing those back out to workers.  \r\n\r\nDask's existing task scheduling model is not ideal for this.  Every computation or communication checks in with the scheduler, which adds unnecessary network roundtrips and provides a central bottleneck for metadata tracking.  To do parameter servers well we probably need to break out of this model and engage more peer-to-peer coordination, without frequent checking in with the scheduler.\r\n\r\nBuilding this infrastructure is not hard, but if possible it would be good to build general machinery that can be used to solve both this problem, and potentially others.  I would appreciate having a conversation with a few people to identify what programming and networking constructs might suffice for this.\r\n\r\ncc @stsievert @fabianp @mlnick"},"comment":{"url":"https://api.github.com/repos/dask/dask-ml/issues/comments/411451595","html_url":"https://github.com/dask/dask-ml/issues/171#issuecomment-411451595","issue_url":"https://api.github.com/repos/dask/dask-ml/issues/171","id":411451595,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTQ1MTU5NQ==","user":{"login":"stsievert","id":1320475,"node_id":"MDQ6VXNlcjEzMjA0NzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1320475?v=4","gravatar_id":"","url":"https://api.github.com/users/stsievert","html_url":"https://github.com/stsievert","followers_url":"https://api.github.com/users/stsievert/followers","following_url":"https://api.github.com/users/stsievert/following{/other_user}","gists_url":"https://api.github.com/users/stsievert/gists{/gist_id}","starred_url":"https://api.github.com/users/stsievert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stsievert/subscriptions","organizations_url":"https://api.github.com/users/stsievert/orgs","repos_url":"https://api.github.com/users/stsievert/repos","events_url":"https://api.github.com/users/stsievert/events{/privacy}","received_events_url":"https://api.github.com/users/stsievert/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T15:39:56Z","updated_at":"2018-08-08T15:39:56Z","author_association":"CONTRIBUTOR","body":"I've timed the parameter server in the notebook [Centralized-PS.ipynb@a971d2][1]. The parameter server behind this graph implements the \"synchronous communication with backup workers\", but without any backup workers (i.e, workers never wait and can repeat gradient computations).\r\n\r\nHere's a quick summary from that notebook:\r\n\r\n| Total optimization time | Optimization time components |\r\n|:---:|:---:|\r\n| ![](https://user-images.githubusercontent.com/1320475/43845618-eb8c84ee-9b1b-11e8-8f72-23fca30c8b97.png) |  ![](https://user-images.githubusercontent.com/1320475/43845662-ffe6e5d8-9b1b-11e8-9981-50800fa38ae8.png) /> |\r\n\r\nThe solid lines are the median, the shaded color borders specify the 95% confidence interval. I get this on my local machine (16GB RAM, no GPU) when timing the MNIST PyTorch example (which is unrealistic: gradients take a long time to compute but the model is small).\r\n\r\n[1]:https://github.com/stsievert/dask-ps/blob/a971d2bd3a6d3b5b9db4824c31e002f2e9c36171/Centralized-PS.ipynb"}}
{"actor":{"display_login":"stsievert"}}