{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/8182","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/8182/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/8182/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/8182/events","html_url":"https://github.com/pytorch/pytorch/pull/8182","id":329692834,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyODg3MTMx","number":8182,"title":"[c10d] NCCL Process Group implementation","user":{"login":"teng-li","id":8120856,"node_id":"MDQ6VXNlcjgxMjA4NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8120856?v=4","gravatar_id":"","url":"https://api.github.com/users/teng-li","html_url":"https://github.com/teng-li","followers_url":"https://api.github.com/users/teng-li/followers","following_url":"https://api.github.com/users/teng-li/following{/other_user}","gists_url":"https://api.github.com/users/teng-li/gists{/gist_id}","starred_url":"https://api.github.com/users/teng-li/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teng-li/subscriptions","organizations_url":"https://api.github.com/users/teng-li/orgs","repos_url":"https://api.github.com/users/teng-li/repos","events_url":"https://api.github.com/users/teng-li/events{/privacy}","received_events_url":"https://api.github.com/users/teng-li/received_events","type":"User","site_admin":false},"labels":[{"id":679953883,"node_id":"MDU6TGFiZWw2Nzk5NTM4ODM=","url":"https://api.github.com/repos/pytorch/pytorch/labels/distributed","name":"distributed","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-06T02:57:38Z","updated_at":"2018-06-07T23:11:08Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/pytorch/pytorch/pulls/8182","html_url":"https://github.com/pytorch/pytorch/pull/8182","diff_url":"https://github.com/pytorch/pytorch/pull/8182.diff","patch_url":"https://github.com/pytorch/pytorch/pull/8182.patch"},"body":"This PR implements a bare-bone NCCL process group. \r\n\r\nBuild: will automatically detect NCCL and NCCL's version as well to decide if it should build with NCCL PG or without NCCL pg.\r\n```\r\n$ cmake ../ -DCMAKE_INSTALL_PREFIX=\"$PWD/../../tmp_install\"\r\n-- The C compiler identification is GNU 5.4.0\r\n-- The CXX compiler identification is GNU 5.4.0\r\n-- Check for working C compiler: /usr/bin/cc\r\n-- Check for working C compiler: /usr/bin/cc -- works\r\n-- Detecting C compiler ABI info\r\n-- Detecting C compiler ABI info - done\r\n-- Detecting C compile features\r\n-- Detecting C compile features - done\r\n-- Check for working CXX compiler: /usr/bin/c++\r\n-- Check for working CXX compiler: /usr/bin/c++ -- works\r\n-- Detecting CXX compiler ABI info\r\n-- Detecting CXX compiler ABI info - done\r\n-- Detecting CXX compile features\r\n-- Detecting CXX compile features - done\r\n-- ATEN_INCLUDE_DIR: /private/home/tengli/pieter_pytorch/pytorch/torch/lib/tmp_install/include\r\n-- ATEN_LIBRARIES: /private/home/tengli/pieter_pytorch/pytorch/torch/lib/tmp_install/lib/libATen_cpu.so;/private/home/tengli/pieter_pytorch/pytorch/torch/lib/tmp_install/lib/libATen_cuda.so\r\n-- Found MPI_C: /public/apps/openmpi/2.1.1/gcc.5.4.0/lib/libmpi.so\r\n-- Found MPI_CXX: /public/apps/openmpi/2.1.1/gcc.5.4.0/lib/libmpi.so\r\n-- MPI_INCLUDE_PATH: /public/apps/openmpi/2.1.1/gcc.5.4.0/include\r\n-- MPI_LIBRARIES: /public/apps/openmpi/2.1.1/gcc.5.4.0/lib/libmpi.so\r\n-- MPIEXEC: /public/apps/openmpi/2.1.1/gcc.5.4.0/bin/mpiexec\r\n-- Found NCCL: /public/apps/NCCL/2.2.12-1/include\r\n-- Determining NCCL version from the header file: /public/apps/NCCL/2.2.12-1/include/nccl.h\r\n-- NCCL_MAJOR_VERSION: 2\r\n-- Found NCCL (include: /public/apps/NCCL/2.2.12-1/include, library: /public/apps/NCCL/2.2.12-1/lib/libnccl.so)\r\n-- NCCL_LIBRARIES: /public/apps/NCCL/2.2.12-1/lib/libnccl.so\r\n-- NCCL_INCLUDE_DIR: /public/apps/NCCL/2.2.12-1/include\r\n-- NCCL Version 2 or higher found, will compile with NCCL distributed backend\r\n-- Looking for pthread.h\r\n-- Looking for pthread.h - found\r\n-- Looking for pthread_create\r\n-- Looking for pthread_create - not found\r\n-- Looking for pthread_create in pthreads\r\n-- Looking for pthread_create in pthreads - not found\r\n-- Looking for pthread_create in pthread\r\n-- Looking for pthread_create in pthread - found\r\n-- Found Threads: TRUE\r\n-- Found CUDA: /public/apps/cuda/9.0 (found version \"9.0\")\r\n-- Configuring done\r\n-- Generating done\r\n-- Build files have been written to: /private/home/tengli/pieter_pytorch/pytorch/torch/lib/c10d/build\r\n```\r\nTesting: ProcessGroupNCCLTest can be either used for single node or multi-node testing. By default, make test will run single node testing:\r\n\r\nTested on 2x DGX1s with 8x GPUs:\r\n**(1) Single Node:**\r\n```\r\n$./ProcessGroupNCCLTest\r\nAllreduce test successful\r\nBroadcast test successful\r\n```\r\n**(2) Multi-node:**\r\n**node0:**\r\n```\r\n$ TMPFILE=\"/private/home/tengli/temp/tengli-test\" RANK=0 WORLD_SIZE=2 ./ProcessGroupNCCLTest\r\nMulti-node world size: 2 rank: 0\r\nAllreduce test successful\r\nBroadcast test successful\r\n```\r\n**node1:**\r\n```\r\n$ TMPFILE=\"/private/home/tengli/temp/tengli-test\" RANK=1 WORLD_SIZE=2 ./ProcessGroupNCCLTest\r\nMulti-node world size: 2 rank: 1\r\nAllreduce test successful\r\nBroadcast test successful\r\n```"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/395594186","html_url":"https://github.com/pytorch/pytorch/pull/8182#issuecomment-395594186","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/8182","id":395594186,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NTU5NDE4Ng==","user":{"login":"teng-li","id":8120856,"node_id":"MDQ6VXNlcjgxMjA4NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8120856?v=4","gravatar_id":"","url":"https://api.github.com/users/teng-li","html_url":"https://github.com/teng-li","followers_url":"https://api.github.com/users/teng-li/followers","following_url":"https://api.github.com/users/teng-li/following{/other_user}","gists_url":"https://api.github.com/users/teng-li/gists{/gist_id}","starred_url":"https://api.github.com/users/teng-li/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teng-li/subscriptions","organizations_url":"https://api.github.com/users/teng-li/orgs","repos_url":"https://api.github.com/users/teng-li/repos","events_url":"https://api.github.com/users/teng-li/events{/privacy}","received_events_url":"https://api.github.com/users/teng-li/received_events","type":"User","site_admin":false},"created_at":"2018-06-07T23:11:08Z","updated_at":"2018-06-07T23:11:08Z","author_association":"CONTRIBUTOR","body":"@pietern All comments addressed"}}
{"actor":{"display_login":"teng-li"}}