{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/274","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/274/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/274/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/274/events","html_url":"https://github.com/uber/horovod/issues/274","id":324593007,"number":274,"title":"Horovod runtime problem mpi_lib.so: undefined symbol: __cudaPopCallConfiguration","user":{"login":"adavanisanti","id":7530528,"avatar_url":"https://avatars0.githubusercontent.com/u/7530528?v=4","gravatar_id":"","url":"https://api.github.com/users/adavanisanti","html_url":"https://github.com/adavanisanti","followers_url":"https://api.github.com/users/adavanisanti/followers","following_url":"https://api.github.com/users/adavanisanti/following{/other_user}","gists_url":"https://api.github.com/users/adavanisanti/gists{/gist_id}","starred_url":"https://api.github.com/users/adavanisanti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adavanisanti/subscriptions","organizations_url":"https://api.github.com/users/adavanisanti/orgs","repos_url":"https://api.github.com/users/adavanisanti/repos","events_url":"https://api.github.com/users/adavanisanti/events{/privacy}","received_events_url":"https://api.github.com/users/adavanisanti/received_events","type":"User","site_admin":false},"labels":[{"id":665452437,"url":"https://api.github.com/repos/uber/horovod/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-05-19T02:09:44Z","updated_at":"2018-05-21T23:27:08Z","closed_at":null,"author_association":"NONE","body":"I have successfully installed horovod using the instructions. But when I try to run any example, I see the following error:\r\n\r\n`import horovod.tensorflow  as hvd`\r\n\r\nError:\r\n```\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/home/ubuntu/horovod/local/lib/python2.7/site-packages/horovod/tensorflow/__init__.py\", line 32, in <module>\r\n    from horovod.common import init\r\n  File \"/home/ubuntu/horovod/local/lib/python2.7/site-packages/horovod/common/__init__.py\", line 47, in <module>\r\n    'mpi_lib' + get_ext_suffix()), mode=ctypes.RTLD_GLOBAL)\r\n  File \"/usr/lib/python2.7/ctypes/__init__.py\", line 362, in __init__\r\n    self._handle = _dlopen(self._name, mode)\r\nOSError: /home/ubuntu/horovod/local/lib/python2.7/site-packages/horovod/common/mpi_lib.so: undefined symbol: __cudaPopCallConfiguration\r\n```\r\n\r\nEnvironment:\r\n\r\nUbuntu 1604\r\nCUDA 9.0\r\nPython 2.7\r\nTensorflow 1.8.0\r\nMPI 3.1.0\r\n\r\nAny pointers on where to look?\r\n\r\nThanks\r\nSanti"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/390813445","html_url":"https://github.com/uber/horovod/issues/274#issuecomment-390813445","issue_url":"https://api.github.com/repos/uber/horovod/issues/274","id":390813445,"user":{"login":"adavanisanti","id":7530528,"avatar_url":"https://avatars0.githubusercontent.com/u/7530528?v=4","gravatar_id":"","url":"https://api.github.com/users/adavanisanti","html_url":"https://github.com/adavanisanti","followers_url":"https://api.github.com/users/adavanisanti/followers","following_url":"https://api.github.com/users/adavanisanti/following{/other_user}","gists_url":"https://api.github.com/users/adavanisanti/gists{/gist_id}","starred_url":"https://api.github.com/users/adavanisanti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adavanisanti/subscriptions","organizations_url":"https://api.github.com/users/adavanisanti/orgs","repos_url":"https://api.github.com/users/adavanisanti/repos","events_url":"https://api.github.com/users/adavanisanti/events{/privacy}","received_events_url":"https://api.github.com/users/adavanisanti/received_events","type":"User","site_admin":false},"created_at":"2018-05-21T23:27:08Z","updated_at":"2018-05-21T23:27:08Z","author_association":"NONE","body":"@alsrgv Reinstalled everything from scratch and it is working now on a single GPU machine. I think the problem was with running.\r\n\r\n`sudo apt install libnccl2 libnccl-dev`\r\ninstead of\r\n`sudo apt install libnccl2=2.2.12-1+cuda9.0 libnccl-dev=2.2.12-1+cuda9.0`"}}
{"actor":{"display_login":"adavanisanti"}}