{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/12527","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/12527/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/12527/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/12527/events","html_url":"https://github.com/pytorch/pytorch/issues/12527","id":368589266,"node_id":"MDU6SXNzdWUzNjg1ODkyNjY=","number":12527,"title":"Failed to deploy distributed pytorch in clusters with dockers using \"bridge network\"","user":{"login":"qjfytz","id":33013591,"node_id":"MDQ6VXNlcjMzMDEzNTkx","avatar_url":"https://avatars2.githubusercontent.com/u/33013591?v=4","gravatar_id":"","url":"https://api.github.com/users/qjfytz","html_url":"https://github.com/qjfytz","followers_url":"https://api.github.com/users/qjfytz/followers","following_url":"https://api.github.com/users/qjfytz/following{/other_user}","gists_url":"https://api.github.com/users/qjfytz/gists{/gist_id}","starred_url":"https://api.github.com/users/qjfytz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qjfytz/subscriptions","organizations_url":"https://api.github.com/users/qjfytz/orgs","repos_url":"https://api.github.com/users/qjfytz/repos","events_url":"https://api.github.com/users/qjfytz/events{/privacy}","received_events_url":"https://api.github.com/users/qjfytz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-10T10:04:00Z","updated_at":"2018-10-11T00:44:07Z","closed_at":null,"author_association":"NONE","body":"\r\n<!-- A clear and concise description of what the bug is. -->\r\n\r\nWe are working on deploying pytorch in a GPU cluster.  But find the distributed process group cannot be established. \r\n\r\nSay we have two machines and in each machine we create a docker where pytorch runs in. Since the network mode is \"bridge mode\"  for the dockers, the process in docker of machine A cannot directly connect to that in docker of machine B. It makes the initialization of process group failed.\r\n\r\nIf we change the network mode to 'host mode', everything is OK. But we have to use the \"bridge mode\". \r\n\r\n\r\n<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->\r\n\r\n## Environment\r\n\r\nbackend: gloo\r\nScript: the script is downloaded from https://github.com/pytorch/examples/tree/master/imagenet\r\nDistributed mode:  we use DistributedDataParallel and each gpu is governed by a process.\r\n```\r\n\r\n - PyTorch Version (e.g., 1.0): 0.4.1\r\n - OS (e.g., Linux): ubuntu 16.04.4\r\n - How you installed PyTorch (`conda`, `pip`, source): pip\r\n - Build command you used (if compiling from source):\r\n - Python version: 3.6\r\n - CUDA/cuDNN version: 9.0/7.1.2\r\n - GPU models and configuration:\r\n - Any other relevant information:\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/428779363","html_url":"https://github.com/pytorch/pytorch/issues/12527#issuecomment-428779363","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/12527","id":428779363,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODc3OTM2Mw==","user":{"login":"teng-li","id":8120856,"node_id":"MDQ6VXNlcjgxMjA4NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8120856?v=4","gravatar_id":"","url":"https://api.github.com/users/teng-li","html_url":"https://github.com/teng-li","followers_url":"https://api.github.com/users/teng-li/followers","following_url":"https://api.github.com/users/teng-li/following{/other_user}","gists_url":"https://api.github.com/users/teng-li/gists{/gist_id}","starred_url":"https://api.github.com/users/teng-li/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teng-li/subscriptions","organizations_url":"https://api.github.com/users/teng-li/orgs","repos_url":"https://api.github.com/users/teng-li/repos","events_url":"https://api.github.com/users/teng-li/events{/privacy}","received_events_url":"https://api.github.com/users/teng-li/received_events","type":"User","site_admin":false},"created_at":"2018-10-11T00:44:07Z","updated_at":"2018-10-11T00:44:07Z","author_association":"CONTRIBUTOR","body":"Two notes from your experimental setup:\r\n\r\n(1), if the two machines share the same shared file system, please use `file://` init method while we will be fixing the `tcp://` init method.\r\n\r\n(2), if you are using multi-processing distributed training and each GPU is governed by a process,  you are recommended to use NCCL backend for this purpose, since it is optimized for such multi-process scenario with CUDA IPC and can be a lot faster."}}
{"actor":{"display_login":"teng-li"}}