{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2626","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2626/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2626/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2626/events","html_url":"https://github.com/tensorflow/tensorflow/issues/2626","id":158230114,"number":2626,"title":"Profiling: libcupti.so cannot be loaded","user":{"login":"wookayin","id":1009873,"avatar_url":"https://avatars.githubusercontent.com/u/1009873?v=3","gravatar_id":"","url":"https://api.github.com/users/wookayin","html_url":"https://github.com/wookayin","followers_url":"https://api.github.com/users/wookayin/followers","following_url":"https://api.github.com/users/wookayin/following{/other_user}","gists_url":"https://api.github.com/users/wookayin/gists{/gist_id}","starred_url":"https://api.github.com/users/wookayin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wookayin/subscriptions","organizations_url":"https://api.github.com/users/wookayin/orgs","repos_url":"https://api.github.com/users/wookayin/repos","events_url":"https://api.github.com/users/wookayin/events{/privacy}","received_events_url":"https://api.github.com/users/wookayin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20response","name":"awaiting response","color":"5319e7"},{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/installation/startup","name":"installation/startup","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-02T20:18:12Z","updated_at":"2016-06-03T18:45:03Z","closed_at":null,"body":"### Environment info\r\nOperating System: Linux Ubuntu 14.04 LTS (64bit)\r\n\r\nInstalled version of CUDA and cuDNN: CUDA 7.5.18 and CUDNN 4.0.7\r\n(please attach the output of `ls -l /path/to/cuda/lib/libcud*`):\r\n\r\n```\r\n❯❯❯ ls -l /usr/local/cuda-7.5/lib/libcud*\r\n-rw----r-- 1 root root 189170  2월 24 18:12 /usr/local/cuda-7.5/lib/libcudadevrt.a\r\nlrwxrwxrwx 1 root root     16  2월 24 18:12 /usr/local/cuda-7.5/lib/libcudart.so -> libcudart.so.7.5\r\nlrwxrwxrwx 1 root root     19  2월 24 18:12 /usr/local/cuda-7.5/lib/libcudart.so.7.5 -> libcudart.so.7.5.18\r\n-rwx---r-x 1 root root 311596  2월 24 18:12 /usr/local/cuda-7.5/lib/libcudart.so.7.5.18\r\n-rw----r-- 1 root root 558020  2월 24 18:12 /usr/local/cuda-7.5/lib/libcudart_static.a\r\n```\r\n```\r\n❯❯❯ ls -l /usr/local/cuda-7.5/lib64/libcud*\r\n-rw----r-- 1 root root   322936  2월 24 18:12 /usr/local/cuda-7.5/lib64/libcudadevrt.a\r\nlrwxrwxrwx 1 root root       16  2월 24 18:12 /usr/local/cuda-7.5/lib64/libcudart.so -> libcudart.so.7.5\r\nlrwxrwxrwx 1 root root       19  2월 24 18:12 /usr/local/cuda-7.5/lib64/libcudart.so.7.5 -> libcudart.so.7.5.18\r\n-rwx---r-x 1 root root   383336  2월 24 18:12 /usr/local/cuda-7.5/lib64/libcudart.so.7.5.18\r\n-rw----r-- 1 root root   720192  2월 24 18:12 /usr/local/cuda-7.5/lib64/libcudart_static.a\r\nlrwxrwxrwx 1 root root       13  3월  3 03:30 /usr/local/cuda-7.5/lib64/libcudnn.so -> libcudnn.so.4\r\nlrwxrwxrwx 1 root root       17  3월  3 03:30 /usr/local/cuda-7.5/lib64/libcudnn.so.4 -> libcudnn.so.4.0.7\r\n-rwxr-xr-x 1 root root 61453024  3월  3 03:30 /usr/local/cuda-7.5/lib64/libcudnn.so.4.0.7\r\n```\r\n\r\nIf installed from binary pip package, provide:\r\n\r\n1. Which pip package you installed : **Tensorflow 0.8.0 Nightly** Python2.7 Linux (GPU) e.g. [Build 118](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-linux/118/)\r\n2. The output from python -c \"import tensorflow; print(tensorflow.__version__)\". : 0.8.0\r\n\r\nIf installed from sources, provide the commit hash:\r\n\r\n### Steps to reproduce\r\n\r\nAlthough it is experimental, I am using the GPU profiling functionality with CUPTI. \r\n\r\n1. Run any tensorflow code that uses CUPTI or `tf.RunOptions.FULL_TRACE`.\r\n2. The following error (segfault) occurs.\r\n\r\n```\r\nI tensorflow/stream_executor/dso_loader.cc:102] Couldn't open CUDA library libcupti.so. LD_LIBRARY_PATH:\r\nF tensorflow/core/common_runtime/gpu/cupti_wrapper.cc:57] Check failed: f != nullptr could not find cuptiActivityRegisterCallbacksin libcupti DSO; dlerror: /home/wookayin/.virtualenvs/tfdebug/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so: undefined symbol: cuptiActivityRegisterCallbacks\r\n[1]    82604 abort (core dumped)  python 19-mnist-profiling.py\r\n```\r\n\r\nAny TF code that invokes loading of `libcupti.so` will face the same error, but for convenience I will share a code that can run standalone: [19-mnist-profiling.py](https://gist.github.com/wookayin/06631c68bb48fc1d0a4eee77cbbba5f1)\r\n\r\n### What have you tried?\r\nThe problem is that the shared library `libcupti.so` cannot be loaded. However, in some older nightly version (such as [Build 103](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-linux/103/)) it worked.\r\n\r\nUPD: I binary-searched to find the changeset to be blamed. Build 103 works, but [Build 104](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-linux/104/) (Failed) and [Build 105](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-linux/105/) does not work.\r\nI highly suspect that this regression is since commit 6bd964c (but not sure):\r\n\r\n- The path to `libcupti.so` would be `/usr/local/cuda/extras/CUPTI/lib64/libcupti.so`.\r\n- After this commit, it seems that path to `libcupti.so` goes wrong. (but why?)\r\n\r\nA strange thing to me is that tensorflow already has [a unit test](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/client/timeline_test.py) for CUPTI and GPU tracing functionalities, so CI must have run this test as well. This bug might be happening in some environments only (like nightly build I installed via `pip`), or it can be a a bazel-related problem (when generating packages).\r\n\r\nI have not investigated into this problem in detail; it looks that after some troubleshooting I can figure out what the cause is.\r\n\r\nThanks!\r\n\r\n### Logs or other output that would be helpful\r\n\r\nN/A"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/223660894","html_url":"https://github.com/tensorflow/tensorflow/issues/2626#issuecomment-223660894","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2626","id":223660894,"user":{"login":"prb12","id":11547801,"avatar_url":"https://avatars.githubusercontent.com/u/11547801?v=3","gravatar_id":"","url":"https://api.github.com/users/prb12","html_url":"https://github.com/prb12","followers_url":"https://api.github.com/users/prb12/followers","following_url":"https://api.github.com/users/prb12/following{/other_user}","gists_url":"https://api.github.com/users/prb12/gists{/gist_id}","starred_url":"https://api.github.com/users/prb12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prb12/subscriptions","organizations_url":"https://api.github.com/users/prb12/orgs","repos_url":"https://api.github.com/users/prb12/repos","events_url":"https://api.github.com/users/prb12/events{/privacy}","received_events_url":"https://api.github.com/users/prb12/received_events","type":"User","site_admin":false},"created_at":"2016-06-03T18:45:03Z","updated_at":"2016-06-03T18:45:03Z","body":"I'm not sure that the change to 'with_the_same_user' is relevant? \r\n(AFAIK, This is just used in our Jenkins test framework to ensure the LD_LIBRARY_PATH from the Docker container is used when running GPU tests as the user 'CI_BUILD_UID')\r\n\r\nNote: the same GIT commit also changed the interpretation of the trace level enum.  Prior to this change, the TRACE_FULL option was only recording the host-side enqueueing of ops (i.e. what is now down if you specify 'SOFTWARE_TRACE') \r\n\r\nIt is now the case the 'TRACE_FULL' will also try to enable the GPUTracer on CUDA builds.  \r\nIf you're running a python script, this requires libcupti to be on your LD_LIBRARY_PATH (since there's no other way to find the NVidia library.)\r\n\r\ne.g. LD_LIBRARY_PATH=/usr/loca/cuda/lib64:/usrlocal/cuda/extras/lib64\r\n\r\n\r\n"}}
{"actor":{"display_login":"prb12"}}