{"action":"created","issue":{"url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/587","repository_url":"https://api.github.com/repos/jupyterhub/jupyterhub","labels_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/587/labels{/name}","comments_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/587/comments","events_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/587/events","html_url":"https://github.com/jupyterhub/jupyterhub/issues/587","id":156857301,"number":587,"title":"Enable MPI from DockerSpawned Jupyter","user":{"login":"robnagler","id":5495179,"avatar_url":"https://avatars.githubusercontent.com/u/5495179?v=3","gravatar_id":"","url":"https://api.github.com/users/robnagler","html_url":"https://github.com/robnagler","followers_url":"https://api.github.com/users/robnagler/followers","following_url":"https://api.github.com/users/robnagler/following{/other_user}","gists_url":"https://api.github.com/users/robnagler/gists{/gist_id}","starred_url":"https://api.github.com/users/robnagler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robnagler/subscriptions","organizations_url":"https://api.github.com/users/robnagler/orgs","repos_url":"https://api.github.com/users/robnagler/repos","events_url":"https://api.github.com/users/robnagler/events{/privacy}","received_events_url":"https://api.github.com/users/robnagler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-25T21:43:15Z","updated_at":"2016-05-26T09:42:52Z","closed_at":null,"body":"I am probably missing something about Jupyter/Hub so I apologize in advance for the rather long explanation and suggestion.\r\n\r\nWe'd like to be able to start MPI from a Jupyter container, which was started by JupyterHub. I've proven that it's possible, but it required quite a few workarounds due to how JupyterHub acquires the IP and port for the Jupyter container.\r\n\r\nIn order to enable MPI, the container must be started with network_mode=host (--net=host). MPI requires an arbitrary set of ports be open. Both the slaves and the master listen on TCP sockets.\r\n\r\nWhen network_mode=host is set, the Jupyter container is sharing ports with all other processes on the host. Therefore, the Jupyter container needs to select a random port to use. The way this works with LocalSpawner is that it opens a random port, closes it, and then passes that port number to the Jupyter process. There's a race condition with this algorithm. With DockerSpawner and network_mode=host, this algorithm can't work, because the spawned containers may not be on the local machine and/or the DockerSpawner may not be running in a network-privileged container.\r\n\r\nDocker's inspect API returns no NetworkSettings when the container is using the host's network. Docker simply has not control over the network so it doesn't know what to report. This is not necessarily a problem except that DockerSpawner has to return an ip and port to JupyterHub, and DockerSpawner.get_ip_and_port expects the Docker API to report the network configuration.\r\n\r\nOne workaround is to use Docker's exec API to talk to the container and get it to dump its network settings. For a simple network configuration, this isn't a problem: you can interrogate /proc/net/tcp. However, for a complex network, that's not so easy. You don't really know which interface will be used to connect the JupyterHub/Proxy to Jupyter -- even then, technically, the routes might not be symmetric, but we can skip this case. :)\r\n\r\nAs a relevant aside, I have noticed that when a Jupyter container has problems starting, it may be \"up\" for long enough for JupyterHub to connect to the container's proxy before Jupyter dies. This is another race condition, because JuypterHub has to poll Jupyter instead of Jupyter notifying JupyterHub all is ok until it polls to see if the container is ok.\r\n\r\nMy suggestion is to have Jupyter notify JupyterHub that it's ok using the Hub API. When that notification happens, JupyterHub will know that Jupyter is ready to serve requests and from which IP/port Jupyter is listening on. This would eliminate the two race conditions as well as replace the polling with a simple timer to handle the case Jupyter doesn't come up. Jupyter would know that it needed to callback to the Hub via an environment variable.\r\n\r\nDoes this seem like a reasonable proposal?\r\n"},"comment":{"url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/comments/221825802","html_url":"https://github.com/jupyterhub/jupyterhub/issues/587#issuecomment-221825802","issue_url":"https://api.github.com/repos/jupyterhub/jupyterhub/issues/587","id":221825802,"user":{"login":"minrk","id":151929,"avatar_url":"https://avatars.githubusercontent.com/u/151929?v=3","gravatar_id":"","url":"https://api.github.com/users/minrk","html_url":"https://github.com/minrk","followers_url":"https://api.github.com/users/minrk/followers","following_url":"https://api.github.com/users/minrk/following{/other_user}","gists_url":"https://api.github.com/users/minrk/gists{/gist_id}","starred_url":"https://api.github.com/users/minrk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/minrk/subscriptions","organizations_url":"https://api.github.com/users/minrk/orgs","repos_url":"https://api.github.com/users/minrk/repos","events_url":"https://api.github.com/users/minrk/events{/privacy}","received_events_url":"https://api.github.com/users/minrk/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T09:42:52Z","updated_at":"2016-05-26T09:42:52Z","body":"> Does this seem like a reasonable proposal?\r\n\r\nYes, absolutely. I proposed something similar to this in #302, but haven't gotten around to it. It will be useful to take DockerSpawner as a use case when trying this out, because there are two distinct cases:\r\n\r\n1. host networking, where the best option is for the SU server to pick its port and notify\r\n2. docker networking, where the SU server *can't* know its own port information wrt the Hub, and the current logic is preferable.\r\n\r\nAs an aside, are you exposing host MPI, or MPI across docker containers? If it is across docker containers, could overlay networks avoid the need for host networking?"}}
{"actor":{"display_login":"minrk"}}