{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/2268","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/2268/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/2268/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/2268/events","html_url":"https://github.com/dmlc/mxnet/issues/2268","id":157343714,"number":2268,"title":"MXNet on Spark Roadmap","user":{"login":"javelinjs","id":631860,"avatar_url":"https://avatars.githubusercontent.com/u/631860?v=3","gravatar_id":"","url":"https://api.github.com/users/javelinjs","html_url":"https://github.com/javelinjs","followers_url":"https://api.github.com/users/javelinjs/followers","following_url":"https://api.github.com/users/javelinjs/following{/other_user}","gists_url":"https://api.github.com/users/javelinjs/gists{/gist_id}","starred_url":"https://api.github.com/users/javelinjs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javelinjs/subscriptions","organizations_url":"https://api.github.com/users/javelinjs/orgs","repos_url":"https://api.github.com/users/javelinjs/repos","events_url":"https://api.github.com/users/javelinjs/events{/privacy}","received_events_url":"https://api.github.com/users/javelinjs/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dmlc/mxnet/labels/call%20for%20contribution","name":"call for contribution","color":"159818"},{"url":"https://api.github.com/repos/dmlc/mxnet/labels/Roadmap","name":"Roadmap","color":"fbca04"},{"url":"https://api.github.com/repos/dmlc/mxnet/labels/Scala-Package","name":"Scala-Package","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-05-28T15:28:29Z","updated_at":"2016-05-31T16:53:37Z","closed_at":null,"body":"https://github.com/dmlc/mxnet/pull/2256 makes MXNet on Spark possible. It works on a stable Spark cluster, but when it is brought to a complex environment, e.g., executors may fail and retry, multiple tasks may configured to run in one executor, etc.\r\n\r\nRelated to issue https://github.com/dmlc/mxnet/issues/1637 @tqchen \r\n\r\nHere's a roadmap for all those issues which may prevent from using MXNet on Spark in production environment.\r\n\r\n- [ ] KVStore workers (and scheduler/servers) failover. I found that, when a worker fails and restarts, it is not able to connect the scheduler again. Is it expected for ps-lite? @mli \r\n- [ ] Timeout for ps scheduler and servers. PS scheduler and servers are started in a spawned process. But when something goes wrong, e.g., workers, servers or the scheduler crash, they will have no chance to stop themselves. I think the simplest way to solve this problem is to set a timeout T, if the scheduler/server does not receive any message in T seconds, stop itself.\r\n- [ ] Multiple ps-lite threads in one process. Currently ps-lite is singleton, but Spark clusters can be configured to run multiple tasks in one executor (related to one process).\r\n- [ ] Make device resources transparent to application. Instead of specifying which GPU to use, users only need to know how many GPUs they want. This is important for clusters which use Yarn to do resource management.\r\n- [ ] Upload and distribute MXNet core library to all the worker nodes automatically."},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/222750415","html_url":"https://github.com/dmlc/mxnet/issues/2268#issuecomment-222750415","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/2268","id":222750415,"user":{"login":"javelinjs","id":631860,"avatar_url":"https://avatars.githubusercontent.com/u/631860?v=3","gravatar_id":"","url":"https://api.github.com/users/javelinjs","html_url":"https://github.com/javelinjs","followers_url":"https://api.github.com/users/javelinjs/followers","following_url":"https://api.github.com/users/javelinjs/following{/other_user}","gists_url":"https://api.github.com/users/javelinjs/gists{/gist_id}","starred_url":"https://api.github.com/users/javelinjs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javelinjs/subscriptions","organizations_url":"https://api.github.com/users/javelinjs/orgs","repos_url":"https://api.github.com/users/javelinjs/repos","events_url":"https://api.github.com/users/javelinjs/events{/privacy}","received_events_url":"https://api.github.com/users/javelinjs/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T16:53:37Z","updated_at":"2016-05-31T16:53:37Z","body":"Hi Mu,\r\n\r\nI think your idea make sense. We do have chance to checkpoint in most use-cases. For the 2nd point, the only way can cause server crash I see is the physical node itself goes out. In such situation, which is rare, we can simply fail the application. But as you said, we do need continuous fault tolerance for\r\nworkers."}}
{"actor":{"display_login":"javelinjs"}}