{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/10960","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/10960/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/10960/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/10960/events","html_url":"https://github.com/pytorch/pytorch/issues/10960","id":354849479,"node_id":"MDU6SXNzdWUzNTQ4NDk0Nzk=","number":10960,"title":"c10d compilation failing","user":{"login":"hanyunfan","id":6924448,"node_id":"MDQ6VXNlcjY5MjQ0NDg=","avatar_url":"https://avatars2.githubusercontent.com/u/6924448?v=4","gravatar_id":"","url":"https://api.github.com/users/hanyunfan","html_url":"https://github.com/hanyunfan","followers_url":"https://api.github.com/users/hanyunfan/followers","following_url":"https://api.github.com/users/hanyunfan/following{/other_user}","gists_url":"https://api.github.com/users/hanyunfan/gists{/gist_id}","starred_url":"https://api.github.com/users/hanyunfan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanyunfan/subscriptions","organizations_url":"https://api.github.com/users/hanyunfan/orgs","repos_url":"https://api.github.com/users/hanyunfan/repos","events_url":"https://api.github.com/users/hanyunfan/events{/privacy}","received_events_url":"https://api.github.com/users/hanyunfan/received_events","type":"User","site_admin":false},"labels":[{"id":778855555,"node_id":"MDU6TGFiZWw3Nzg4NTU1NTU=","url":"https://api.github.com/repos/pytorch/pytorch/labels/build","name":"build","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"pietern","id":9845,"node_id":"MDQ6VXNlcjk4NDU=","avatar_url":"https://avatars3.githubusercontent.com/u/9845?v=4","gravatar_id":"","url":"https://api.github.com/users/pietern","html_url":"https://github.com/pietern","followers_url":"https://api.github.com/users/pietern/followers","following_url":"https://api.github.com/users/pietern/following{/other_user}","gists_url":"https://api.github.com/users/pietern/gists{/gist_id}","starred_url":"https://api.github.com/users/pietern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pietern/subscriptions","organizations_url":"https://api.github.com/users/pietern/orgs","repos_url":"https://api.github.com/users/pietern/repos","events_url":"https://api.github.com/users/pietern/events{/privacy}","received_events_url":"https://api.github.com/users/pietern/received_events","type":"User","site_admin":false},"assignees":[{"login":"pietern","id":9845,"node_id":"MDQ6VXNlcjk4NDU=","avatar_url":"https://avatars3.githubusercontent.com/u/9845?v=4","gravatar_id":"","url":"https://api.github.com/users/pietern","html_url":"https://github.com/pietern","followers_url":"https://api.github.com/users/pietern/followers","following_url":"https://api.github.com/users/pietern/following{/other_user}","gists_url":"https://api.github.com/users/pietern/gists{/gist_id}","starred_url":"https://api.github.com/users/pietern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pietern/subscriptions","organizations_url":"https://api.github.com/users/pietern/orgs","repos_url":"https://api.github.com/users/pietern/repos","events_url":"https://api.github.com/users/pietern/events{/privacy}","received_events_url":"https://api.github.com/users/pietern/received_events","type":"User","site_admin":false},{"login":"teng-li","id":8120856,"node_id":"MDQ6VXNlcjgxMjA4NTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8120856?v=4","gravatar_id":"","url":"https://api.github.com/users/teng-li","html_url":"https://github.com/teng-li","followers_url":"https://api.github.com/users/teng-li/followers","following_url":"https://api.github.com/users/teng-li/following{/other_user}","gists_url":"https://api.github.com/users/teng-li/gists{/gist_id}","starred_url":"https://api.github.com/users/teng-li/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teng-li/subscriptions","organizations_url":"https://api.github.com/users/teng-li/orgs","repos_url":"https://api.github.com/users/teng-li/repos","events_url":"https://api.github.com/users/teng-li/events{/privacy}","received_events_url":"https://api.github.com/users/teng-li/received_events","type":"User","site_admin":false}],"milestone":null,"comments":11,"created_at":"2018-08-28T18:26:27Z","updated_at":"2018-08-30T13:40:34Z","closed_at":null,"author_association":"NONE","body":"## Issue description\r\n\r\nCannot compile successfully on pytorch source code build\r\n\r\n## Code example\r\n\r\nVERBOSE=1 NO_SYSTEM_NCCL=1 USE_GLOO_IBVERBS=1 TORCH_CUDA_ARCH_LIST=\"7.0\" python setup.py install\r\n\r\nAnd it reports:\r\n\r\n[ 66%] Built target FileStoreTest\r\n[ 71%] Linking CXX executable allreduce\r\ncd /home/frank/pytorch/torch/lib/build/c10d/example && /home/frank/miniconda3/bin/cmake -E cmake_link_script CMakeFiles/allreduce.dir/link.txt --verbose=1\r\n[ 76%] Linking CXX executable TCPStoreTest\r\ncd /home/frank/pytorch/torch/lib/build/c10d/test && /home/frank/miniconda3/bin/cmake -E cmake_link_script CMakeFiles/TCPStoreTest.dir/link.txt --verbose=1\r\n/bin/c++   -I\"/home/frank/pytorch/torch/lib/tmp_install/include\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/TH\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THC\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THS\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THCS\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THNN\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THCUNN\" -DOMPI_SKIP_MPICXX=1 -fexceptions  -std=c++11 -O3 -DNDEBUG  -L\"/home/frank/pytorch/torch/lib/tmp_install/lib\"  -Wl,-rpath,$ORIGIN -rdynamic CMakeFiles/allreduce.dir/allreduce.cpp.o  -o allreduce  -L/home/frank/miniconda3/lib -Wl,-rpath,/home/frank/pytorch/torch/lib/tmp_install/lib:/home/frank/miniconda3/lib -lpthread ../libc10d.a /home/frank/pytorch/torch/lib/tmp_install/lib/libcaffe2_gpu.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcudart_static.a -ldl -lrt /home/frank/pytorch/torch/lib/tmp_install/lib/libcaffe2.so /home/frank/pytorch/torch/lib/tmp_install/lib64/libprotobuf.a -lpthread -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -fopenmp -lpthread -lm -ldl /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcufft.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcurand.so /home/rengan/apps/cudnn/7.2.1/lib64/libcudnn.so.7 /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcublas.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcublas_device.a /home/frank/pytorch/torch/lib/tmp_install/lib/libgloo_cuda.so /home/frank/pytorch/torch/lib/tmp_install/lib/libgloo.so\r\n/bin/c++   -I\"/home/frank/pytorch/torch/lib/tmp_install/include\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/TH\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THC\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THS\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THCS\"   -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THNN\" -I\"/home/frank/pytorch/torch/lib/tmp_install/include/THCUNN\" -DOMPI_SKIP_MPICXX=1 -fexceptions  -std=c++11 -O3 -DNDEBUG  -L\"/home/frank/pytorch/torch/lib/tmp_install/lib\"  -Wl,-rpath,$ORIGIN -rdynamic CMakeFiles/TCPStoreTest.dir/TCPStoreTest.cpp.o  -o TCPStoreTest  -L/home/frank/miniconda3/lib -Wl,-rpath,/home/frank/pytorch/torch/lib/tmp_install/lib:/home/frank/miniconda3/lib -lpthread ../libc10d.a /home/frank/pytorch/torch/lib/tmp_install/lib/libcaffe2_gpu.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcudart_static.a -ldl -lrt /home/frank/pytorch/torch/lib/tmp_install/lib/libcaffe2.so /home/frank/pytorch/torch/lib/tmp_install/lib64/libprotobuf.a -lpthread -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -fopenmp -lpthread -lm -ldl /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcufft.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcurand.so /home/rengan/apps/cudnn/7.2.1/lib64/libcudnn.so.7 /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcublas.so /cm/shared/apps/cuda92/toolkit/9.2.88/lib64/libcublas_device.a /home/frank/pytorch/torch/lib/tmp_install/lib/libgloo_cuda.so /home/frank/pytorch/torch/lib/tmp_install/lib/libgloo.so\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `std::_Function_handler<void (), c10d::ProcessGroupGloo::broadcast(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::BroadcastOptions const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&)':\r\nProcessGroupGloo.cpp:(.text+0x1c53): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `std::_Function_handler<void (), c10d::ProcessGroupGloo::allreduce(std::vector<at::Tensor, std::allocator<at::Tensor> >&, c10d::AllreduceOptions const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&)':\r\nProcessGroupGloo.cpp:(.text+0x1f13): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `at::DeviceGuard::set_index(int)':\r\nProcessGroupGloo.cpp:(.text._ZN2at11DeviceGuard9set_indexEi[_ZN2at11DeviceGuard9set_indexEi]+0xcc): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `c10d::THCStreamGuard::~THCStreamGuard()':\r\nProcessGroupGloo.cpp:(.text._ZN4c10d14THCStreamGuardD2Ev[_ZN4c10d14THCStreamGuardD5Ev]+0x144): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `std::vector<at::Tensor, std::allocator<at::Tensor> >::vector(std::vector<at::Tensor, std::allocator<at::Tensor> > const&)':\r\nProcessGroupGloo.cpp:(.text._ZNSt6vectorIN2at6TensorESaIS1_EEC2ERKS3_[_ZNSt6vectorIN2at6TensorESaIS1_EEC5ERKS3_]+0x283): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `gloo::BroadcastOneToAll<float>::BroadcastOneToAll(std::shared_ptr<gloo::Context> const&, std::vector<float*, std::allocator<float*> > const&, unsigned long, int, int)':\r\nProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIfEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIfEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii]+0x3b8): undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)'\r\nProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIfEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIfEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii]+0x4a8): undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)'\r\nProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIfEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIfEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii]+0x882): undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)'\r\nProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIfEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIfEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPfSaIS8_EEmii]+0x90b): undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o): In function `gloo::BroadcastOneToAll<double>::BroadcastOneToAll(std::shared_ptr<gloo::Context> const&, std::vector<double*, std::allocator<double*> > const&, unsigned long, int, int)':\r\nProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIdEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPdSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIdEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPdSaIS8_EEmii]+0x3b8): undefined reference to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)'\r\n../libc10d.a(ProcessGroupGloo.cpp.o):ProcessGroupGloo.cpp:(.text._ZN4gloo17BroadcastOneToAllIdEC2ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPdSaIS8_EEmii[_ZN4gloo17BroadcastOneToAllIdEC5ERKSt10shared_ptrINS_7ContextEERKSt6vectorIPdSaIS8_EEmii]+0x4a8): more undefined references to `gloo::EnforceNotMet::EnforceNotMet(char const*, int, char const*, std::string const&)' follow\r\nCMakeFiles/allreduce.dir/allreduce.cpp.o: In function `void std::vector<at::Tensor, std::allocator<at::Tensor> >::_M_emplace_back_aux<at::Tensor const&>(at::Tensor const&)':\r\nallreduce.cpp:(.text._ZNSt6vectorIN2at6TensorESaIS1_EE19_M_emplace_back_auxIIRKS1_EEEvDpOT_[_ZNSt6vectorIN2at6TensorESaIS1_EE19_M_emplace_back_auxIIRKS1_EEEvDpOT_]+0x271): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\nCMakeFiles/allreduce.dir/allreduce.cpp.o: In function `main':\r\nallreduce.cpp:(.text.startup+0x75e): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\nallreduce.cpp:(.text.startup+0x7e9): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\nallreduce.cpp:(.text.startup+0x995): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\nallreduce.cpp:(.text.startup+0xb14): undefined reference to `at::Error::Error(at::SourceLocation, std::string const&)'\r\nCMakeFiles/allreduce.dir/allreduce.cpp.o:allreduce.cpp:(.text.startup+0xd62): more undefined references to `at::Error::Error(at::SourceLocation, std::string const&)' follow\r\ncollect2: error: ld returned 1 exit status\r\nmake[2]: *** [example/allreduce] Error 1\r\nmake[2]: Leaving directory `/home/frank/pytorch/torch/lib/build/c10d'\r\nmake[1]: *** [example/CMakeFiles/allreduce.dir/all] Error 2\r\nmake[1]: *** Waiting for unfinished jobs....\r\nmake[2]: Leaving directory `/home/frank/pytorch/torch/lib/build/c10d'\r\n[ 76%] Built target TCPStoreTest\r\nGenerated /home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/c10d_cuda_test.dir//./c10d_cuda_test_generated_CUDATest.cu.o successfully.\r\ncd /home/frank/pytorch/torch/lib/build/c10d && /home/frank/miniconda3/bin/cmake -E cmake_depends \"Unix Makefiles\" /home/frank/pytorch/torch/lib/c10d /home/frank/pytorch/torch/lib/c10d/test /home/frank/pytorch/torch/lib/build/c10d /home/frank/pytorch/torch/lib/build/c10d/test /home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/c10d_cuda_test.dir/DependInfo.cmake --color=\r\nDependee \"/home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/c10d_cuda_test.dir/DependInfo.cmake\" is newer than depender \"/home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/c10d_cuda_test.dir/depend.internal\".\r\nDependee \"/home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/CMakeDirectoryInformation.cmake\" is newer than depender \"/home/frank/pytorch/torch/lib/build/c10d/test/CMakeFiles/c10d_cuda_test.dir/depend.internal\".\r\nScanning dependencies of target c10d_cuda_test\r\nmake[2]: Leaving directory `/home/frank/pytorch/torch/lib/build/c10d'\r\nmake -f test/CMakeFiles/c10d_cuda_test.dir/build.make test/CMakeFiles/c10d_cuda_test.dir/build\r\nmake[2]: Entering directory `/home/frank/pytorch/torch/lib/build/c10d'\r\n[ 80%] Linking CXX static library libc10d_cuda_test.a\r\ncd /home/frank/pytorch/torch/lib/build/c10d/test && /home/frank/miniconda3/bin/cmake -P CMakeFiles/c10d_cuda_test.dir/cmake_clean_target.cmake\r\ncd /home/frank/pytorch/torch/lib/build/c10d/test && /home/frank/miniconda3/bin/cmake -E cmake_link_script CMakeFiles/c10d_cuda_test.dir/link.txt --verbose=1\r\n/bin/ar qc libc10d_cuda_test.a  CMakeFiles/c10d_cuda_test.dir/c10d_cuda_test_generated_CUDATest.cu.o\r\n/bin/ranlib libc10d_cuda_test.a\r\nmake[2]: Leaving directory `/home/frank/pytorch/torch/lib/build/c10d'\r\n[ 80%] Built target c10d_cuda_test\r\nmake[1]: Leaving directory `/home/frank/pytorch/torch/lib/build/c10d'\r\nmake: *** [all] Error 2\r\nFailed to run 'bash ../tools/build_pytorch_libs.sh --use-cuda --use-nnpack --use-mkldnn --use-gloo-ibverbs nccl caffe2 libshm gloo THD c10d'\r\n\r\n\r\n## System Info\r\n collect_env.py cannot run, due to torch cannot be loaded.\r\n\r\n- PyTorch or Caffe2: PyTorch\r\n- How you installed PyTorch: source\r\n- Build command you used (if compiling from source): VERBOSE=1 NO_SYSTEM_NCCL=1 USE_GLOO_IBVERBS=1 TORCH_CUDA_ARCH_LIST=\"7.2.0\" python setup.py install\r\n- OS: RHEL 7.5\r\n- PyTorch version: torch-0.5.0a0+de09956\r\n- Python version: 3.5.5 installed from miniconda\r\n- CUDA/cuDNN version: 9.2/7.2.1\r\n- GPU models and configuration: V100 x 4\r\n- GCC version (if compiling from source): tried 4.8.5, 5.5.0 and 7.0. none of them work.\r\n- CMake version: 3.12.0\r\n- Versions of any other relevant libraries: NCCL 2.2.13\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/417323155","html_url":"https://github.com/pytorch/pytorch/issues/10960#issuecomment-417323155","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/10960","id":417323155,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzMyMzE1NQ==","user":{"login":"hanyunfan","id":6924448,"node_id":"MDQ6VXNlcjY5MjQ0NDg=","avatar_url":"https://avatars2.githubusercontent.com/u/6924448?v=4","gravatar_id":"","url":"https://api.github.com/users/hanyunfan","html_url":"https://github.com/hanyunfan","followers_url":"https://api.github.com/users/hanyunfan/followers","following_url":"https://api.github.com/users/hanyunfan/following{/other_user}","gists_url":"https://api.github.com/users/hanyunfan/gists{/gist_id}","starred_url":"https://api.github.com/users/hanyunfan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanyunfan/subscriptions","organizations_url":"https://api.github.com/users/hanyunfan/orgs","repos_url":"https://api.github.com/users/hanyunfan/repos","events_url":"https://api.github.com/users/hanyunfan/events{/privacy}","received_events_url":"https://api.github.com/users/hanyunfan/received_events","type":"User","site_admin":false},"created_at":"2018-08-30T13:40:34Z","updated_at":"2018-08-30T13:40:34Z","author_association":"NONE","body":"@orionr just want you to know, I also tried to compile with FULL_CAFFE2=1 after modifying the file as you mentioned, but also the same.\r\n\r\nerror message:\r\n/home/frank/pytorch/caffe2/operators/given_tensor_byte_string_to_uint8_fill_op.h: In member function ‘void caffe2::GivenTensorByteStringToUInt8FillOp<Context>::Extract()’:\r\n/home/frank/pytorch/caffe2/operators/given_tensor_byte_string_to_uint8_fill_op.h:52:51: error: ‘GetRepeatedArgument<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >’ is not a member of ‘caffe2::GivenTensorByteStringToUInt8FillOp<Context>’\r\n     auto source_values = OperatorBase::GetRepeatedArgument<string>(\"values\");\r\n                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n.....\r\nFailed to run 'bash ../tools/build_pytorch_libs.sh --use-cuda --use-nnpack --use-mkldnn --use-gloo-ibverbs --full-caffe2 nccl caffe2 libshm gloo THD c10d'\r\n\r\n"}}
{"actor":{"display_login":"hanyunfan"}}