{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6698","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6698/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6698/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6698/events","html_url":"https://github.com/tensorflow/tensorflow/issues/6698","id":199245869,"number":6698,"title":"Crash: Could not create cuDNN handle when convnets are used","user":{"login":"ymfa","id":6981180,"avatar_url":"https://avatars1.githubusercontent.com/u/6981180?v=4","gravatar_id":"","url":"https://api.github.com/users/ymfa","html_url":"https://github.com/ymfa","followers_url":"https://api.github.com/users/ymfa/followers","following_url":"https://api.github.com/users/ymfa/following{/other_user}","gists_url":"https://api.github.com/users/ymfa/gists{/gist_id}","starred_url":"https://api.github.com/users/ymfa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymfa/subscriptions","organizations_url":"https://api.github.com/users/ymfa/orgs","repos_url":"https://api.github.com/users/ymfa/repos","events_url":"https://api.github.com/users/ymfa/events{/privacy}","received_events_url":"https://api.github.com/users/ymfa/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false},{"id":473173351,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:build/install","name":"type:build/install","color":"c5def5","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":74,"created_at":"2017-01-06T17:29:09Z","updated_at":"2018-03-07T09:26:58Z","closed_at":"2017-06-16T22:31:40Z","author_association":"NONE","body":"Tensorflow (GPU) was imported successfully, but when running a session that involves a convolutional neural network (CNN), Python crashes with the following message:\r\n\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM\r\n    F tensorflow/core/kernels/conv_ops.cc:605] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms) \r\n\r\nThe problem persists on any combination of CUDA toolkit 7.5/8.0 and Tensorflow installed from pip/source. Test sessions that do not use CNNs are run successfully.\r\n\r\n### What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?\r\n\r\nThe issue is similar to https://github.com/tensorflow/tensorflow/issues/6586, where I first commented. But since I experience the problem on a Mac, I was suggested to open a separate issue.\r\n\r\n### Environment info\r\nOperating System: macOS Sierra 10.12.2\r\nXcode version 8.2 (8C38) (When I later tried CUDA 7.5, I installed Command Line Tools version 7.3.1 because CUDA 7.5 lacked support of the more recent compilers.)\r\nPython 3.5.2 (anaconda)\r\n\r\nInstalled version of CUDA: tried both 8.0 (initially) and 7.5 (reported here, toolkit only -- the driver is still 8.0)\r\nInstalled version of cuDNN: 5.1 (different installations according to CUDA versions)\r\n(please attach the output of `ls -l /path/to/cuda/lib/libcud*`):\r\n\r\n    lrwxr-xr-x  1 root   wheel        33  5 Jan 20:33 /usr/local/cuda/lib/libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib\r\n    -rwxr-xr-x@ 1 root   wheel      8280 13 Apr  2016 /usr/local/cuda/lib/libcuda.dylib\r\n    lrwxr-xr-x@ 1 root   wheel        45 13 Apr  2016 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a\r\n    lrwxr-xr-x@ 1 root   wheel        50 13 Apr  2016 /usr/local/cuda/lib/libcudart.7.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib\r\n    lrwxr-xr-x@ 1 root   wheel        46 13 Apr  2016 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib\r\n    lrwxr-xr-x@ 1 root   wheel        49 13 Apr  2016 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a\r\n    lrwxr-xr-x  1 root   wheel        16  5 Jan 17:14 /usr/local/cuda/lib/libcudnn.5 -> libcudnn.5.dylib\r\n    -rwxr-xr-x@ 1 ymfa   staff  58975112 10 Jun  2016 /usr/local/cuda/lib/libcudnn.5.dylib\r\n    lrwxr-xr-x@ 1 ymfa   staff        16 10 Jun  2016 /usr/local/cuda/lib/libcudnn.dylib -> libcudnn.5.dylib\r\n    lrwxr-xr-x  1 root   wheel        16  5 Jan 17:14 /usr/local/cuda/lib/libcudnn5.dylib -> libcudnn.5.dylib\r\n    -rw-r--r--@ 1 ymfa   staff  56392320 10 Jun  2016 /usr/local/cuda/lib/libcudnn_static.a\r\n\r\nI tried both installing from pip and source. I first installed from binary pip package:\r\n\r\n1. A link to the pip package you installed:\r\n`tensorflow-gpu`\r\n2. The output from `python -c \"import tensorflow; print(tensorflow.__version__)\"`.\r\n`0.12.head`\r\n\r\nLater I installed from source (the pip package was uninstalled):\r\n\r\n1. The commit hash (`git rev-parse HEAD`)\r\n`d67c09d98a576e1fbf2f3609ddb842e53890f31c`\r\n2. The output of `bazel version`\r\n\r\n    Build label: 0.4.3-homebrew\r\n    Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar\r\n    Build time: Thu Dec 22 15:20:15 2016 (1482420015)\r\n    Build timestamp: 1482420015\r\n    Build timestamp as int: 1482420015\r\n\r\n### If possible, provide a minimal reproducible example\r\n\r\nI made a minimal example by simplifying the network and reducing the training data to only twenty images and two classes for classification. [issue.zip](https://github.com/tensorflow/tensorflow/files/691561/issue.zip) contains the Python code and the data. I wrote two convolutional layers because I found the network with only one convolutional layer runs without problem.\r\n\r\n### Complete log using CUDA 7.5 and Tensorflow compiled from source\r\n\r\n    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcublas.7.5.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcudnn.5.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcufft.7.5.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcuda.1.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcurand.7.5.dylib locally\r\n    W tensorflow/core/platform/cpu_feature_guard.cc:95] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\r\n    W tensorflow/core/platform/cpu_feature_guard.cc:95] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\r\n    W tensorflow/core/platform/cpu_feature_guard.cc:95] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\r\n    I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:874] OS X does not support NUMA - returning NUMA node zero\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: \r\n    name: GeForce GT 650M\r\n    major: 3 minor: 0 memoryClockRate (GHz) 0.9\r\n    pciBusID 0000:01:00.0\r\n    Total memory: 1023.69MiB\r\n    Free memory: 740.18MiB\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 \r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y \r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM\r\n    F tensorflow/core/kernels/conv_ops.cc:605] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms) \r\n\r\n### Complete log using CUDA 8.0 and Tensorflow installed from pip\r\n\r\n    I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.1.dylib locally\r\n    I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.dylib locally\r\n    I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] OS X does not support NUMA - returning NUMA node zero\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: \r\n    name: GeForce GT 650M\r\n    major: 3 minor: 0 memoryClockRate (GHz) 0.9\r\n    pciBusID 0000:01:00.0\r\n    Total memory: 1023.69MiB\r\n    Free memory: 590.00MiB\r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 \r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y \r\n    I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:392] error retrieving driver version: Invalid argument: expected %d.%d or %d.%d.%d form for driver version; got \"\"\r\n    E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM\r\n    F tensorflow/core/kernels/conv_ops.cc:532] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/371077136","html_url":"https://github.com/tensorflow/tensorflow/issues/6698#issuecomment-371077136","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6698","id":371077136,"user":{"login":"npeirson","id":20098190,"avatar_url":"https://avatars1.githubusercontent.com/u/20098190?v=4","gravatar_id":"","url":"https://api.github.com/users/npeirson","html_url":"https://github.com/npeirson","followers_url":"https://api.github.com/users/npeirson/followers","following_url":"https://api.github.com/users/npeirson/following{/other_user}","gists_url":"https://api.github.com/users/npeirson/gists{/gist_id}","starred_url":"https://api.github.com/users/npeirson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/npeirson/subscriptions","organizations_url":"https://api.github.com/users/npeirson/orgs","repos_url":"https://api.github.com/users/npeirson/repos","events_url":"https://api.github.com/users/npeirson/events{/privacy}","received_events_url":"https://api.github.com/users/npeirson/received_events","type":"User","site_admin":false},"created_at":"2018-03-07T09:26:58Z","updated_at":"2018-03-07T09:26:58Z","author_association":"NONE","body":"The solution from @strickon and @ggranum plus a driver update resolved this for me."}}
{"actor":{"display_login":"npeirson"}}