{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/2402","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/2402/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/2402/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/2402/events","html_url":"https://github.com/dask/distributed/issues/2402","id":388427831,"node_id":"MDU6SXNzdWUzODg0Mjc4MzE=","number":2402,"title":"Could dask-mpi run the client script too?","user":{"login":"kmpaul","id":11411331,"node_id":"MDQ6VXNlcjExNDExMzMx","avatar_url":"https://avatars3.githubusercontent.com/u/11411331?v=4","gravatar_id":"","url":"https://api.github.com/users/kmpaul","html_url":"https://github.com/kmpaul","followers_url":"https://api.github.com/users/kmpaul/followers","following_url":"https://api.github.com/users/kmpaul/following{/other_user}","gists_url":"https://api.github.com/users/kmpaul/gists{/gist_id}","starred_url":"https://api.github.com/users/kmpaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmpaul/subscriptions","organizations_url":"https://api.github.com/users/kmpaul/orgs","repos_url":"https://api.github.com/users/kmpaul/repos","events_url":"https://api.github.com/users/kmpaul/events{/privacy}","received_events_url":"https://api.github.com/users/kmpaul/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-12-06T22:37:02Z","updated_at":"2018-12-12T15:41:14Z","closed_at":null,"author_association":"NONE","body":"I've been dealing with an issue that...well, I was convinced shouldn't be an issue, so I never said anything about it until dask/dask-blog#5.  And after a discussion with @guillaumeeb, I was convinced that maybe I'm not as crazy (or as ill-informed) as I thought I was.  So, here's the issue...\r\n\r\nI've been trying to figure out a way of launching the Dask Scheduler, Workers, _and_ the Client script in the same MPI environment.  Currently, the way `dask-mpi` works is that the Scheduler and the Workers are started, and you separately connect your client (in your separate script) to the Scheduler via, for example, the `scheduler.json` file.\r\n\r\nI discussed with @guillaumeeb one approach that should work, something like the following:\r\n\r\n```bash\r\n# [PBS header info requesting N MPI processes]\r\n\r\nmpirun -np N dask-mpi [dask-mpi options] &\r\npython my_dask_script.py\r\n```\r\n\r\nHowever, this launches Scheduler/Worker processes on all N allocated MPI processes, and then the `python my_dask_script.py` process could, potentially, run on the same process as the Scheduler, for example.  If you have a compute-intensive client script, this could be problematic.\r\n\r\nWhat I was originally hoping for was a solution that allowed something more like this:\r\n\r\n```bash\r\n# [PBS header info requesting N MPI processes]\r\n\r\nmpirun -np N dask-mpi [dask-mpi options] --script my_dask_script\r\n```\r\n\r\nBut after thinking about it for a while, I found that what I _really_ wanted was something that worked like this:\r\n\r\n```bash\r\n# [PBS header info requesting N MPI processes]\r\n\r\nmpirun -np N python my_dask_mpi_script.py\r\n```\r\n\r\nwhere the `my_dask_mpi_script.py` has something like an `import dask-mpi` line that does the following:\r\n\r\n1. let's MPI rank 0 pass through,\r\n2. launches the Scheduler on MPI rank 1 and runs an `IOLoop` until the rank 0 process is complete,\r\n3. launches the Workers on MPI ranks >1, which also run until rank 0 process is complete.\r\n\r\nAt this point, I feel like I could write this myself...except that I don't know how to implement the \"run the `IOLoop` until rank 0 process is complete\" part.\r\n\r\nAny thoughts?  Are there different solutions?  Would you recommend something different?"},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/446632651","html_url":"https://github.com/dask/distributed/issues/2402#issuecomment-446632651","issue_url":"https://api.github.com/repos/dask/distributed/issues/2402","id":446632651,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjYzMjY1MQ==","user":{"login":"kmpaul","id":11411331,"node_id":"MDQ6VXNlcjExNDExMzMx","avatar_url":"https://avatars3.githubusercontent.com/u/11411331?v=4","gravatar_id":"","url":"https://api.github.com/users/kmpaul","html_url":"https://github.com/kmpaul","followers_url":"https://api.github.com/users/kmpaul/followers","following_url":"https://api.github.com/users/kmpaul/following{/other_user}","gists_url":"https://api.github.com/users/kmpaul/gists{/gist_id}","starred_url":"https://api.github.com/users/kmpaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmpaul/subscriptions","organizations_url":"https://api.github.com/users/kmpaul/orgs","repos_url":"https://api.github.com/users/kmpaul/repos","events_url":"https://api.github.com/users/kmpaul/events{/privacy}","received_events_url":"https://api.github.com/users/kmpaul/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T15:41:14Z","updated_at":"2018-12-12T15:41:14Z","author_association":"NONE","body":"I like the first pattern.  I think it will make more sense to \"traditional\" MPI users.  I wonder how you would feel about dropping the `initialize()` method, though.  It could be initialized at import-time, but I don't know if its a good design.  I like the \"at import-time\" initialization, in part, because if the script is running with `mpirun`, then all of the code is running on multiple MPI ranks from the start.  And, while I know that the `import dask_mpi` line could be anywhere, it is canonically at the top of the script.  Does that make sense?\r\n\r\nI might have time (and I definitely have interest) in doing this today.  I think the missing piece that @mrocklin provided for me was the `client.sync` call when shutting down.  So, the question I have from you all is how should this be packaged?  Should it be a PR to `distributed` or should I make a new repo?"}}
{"actor":{"display_login":"kmpaul"}}