{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/15783","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/15783/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/15783/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/15783/events","html_url":"https://github.com/tensorflow/tensorflow/issues/15783","id":285396750,"number":15783,"title":"CUDA crashes during Cholesky_grad procedure","user":{"login":"tom-a-horrocks","id":6022830,"avatar_url":"https://avatars2.githubusercontent.com/u/6022830?v=4","gravatar_id":"","url":"https://api.github.com/users/tom-a-horrocks","html_url":"https://github.com/tom-a-horrocks","followers_url":"https://api.github.com/users/tom-a-horrocks/followers","following_url":"https://api.github.com/users/tom-a-horrocks/following{/other_user}","gists_url":"https://api.github.com/users/tom-a-horrocks/gists{/gist_id}","starred_url":"https://api.github.com/users/tom-a-horrocks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tom-a-horrocks/subscriptions","organizations_url":"https://api.github.com/users/tom-a-horrocks/orgs","repos_url":"https://api.github.com/users/tom-a-horrocks/repos","events_url":"https://api.github.com/users/tom-a-horrocks/events{/privacy}","received_events_url":"https://api.github.com/users/tom-a-horrocks/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-01-02T10:27:26Z","updated_at":"2018-03-18T16:01:14Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes (see command below)\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10\r\n- **TensorFlow installed from (source or binary)**: Binary (pip)\r\n- **TensorFlow version (use command below)**: 'unknown' 1.4.0\r\n- **Python version**: 3.5.2\r\n- **CUDA/cuDNN version**: Cuda: 8.0.61.2; cuDNN: 6.1\r\n- **GPU model and memory**: GeForce GTX 770M (3GB); NVIDIA driver 388.71\r\n- **Bazel version**: N/A\r\n- **Exact command to reproduce**:\r\n```\r\nimport tensorflow as tf\r\nimport numpy as np\r\n\r\nwith tf.Session() as sess:\r\n    x = tf.placeholder(tf.float64, [None, None])\r\n    f = tf.reduce_sum(tf.cholesky(x))\r\n    sess.run(tf.global_variables_initializer())\r\n    print(sess.run(tf.gradients(f, x), {x:np.array(1.).reshape(1, 1)}))\r\n```\r\n\r\n### Describe the problem\r\nPython crashes when running the code above. It was initially encountered when trying to perform a Cholesky decomposition during sparse Gaussian process regression (see https://github.com/GPflow/GPflow/issues/602). I am out of my depth here, but the following error messages seemed interesting:\r\n* Address 0x00000000 is out of bounds\r\n* Blas GEMV launch failed\r\n* failed to run cuBLAS routine cublasDgemv_v2: CUBLAS_STATUS_EXECUTION_FAILED\r\n\r\nI have run various CUDA profiling tests and have had no issues. The cholesky decomposition example that comes with CUDA could also execute just fine. I have tried both reducing the amount of GPU memory available to tensorflow _and_ setting it grow-able, to no effect.\r\n\r\n### Source code / logs\r\nSee above for source code.\r\n\r\nstdout/stderr:\r\n```\r\n2018-01-02 18:14:34.305556: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\core\\platform\\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2\r\n2018-01-02 18:14:35.110794: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1030] Found device 0 with properties:\r\nname: GeForce GTX 770M major: 3 minor: 0 memoryClockRate(GHz): 0.797\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 3.00GiB freeMemory: 2.50GiB\r\n2018-01-02 18:14:35.110932: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 770M, pci bus id: 0000:01:00.0, compute capability: 3.0)\r\n2018-01-02 18:14:35.290410: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\core\\kernels\\cuda_solvers.cc:159] Creating CudaSolver handles for stream 00000189FC81C670\r\n2018-01-02 18:14:35.615695: E C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\stream_executor\\cuda\\cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS\r\n2018-01-02 18:14:35.615847: F C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\core\\common_runtime\\gpu\\gpu_event_mgr.cc:203] Unexpected Event status: 1\r\n2018-01-02 18:14:35.615695: E C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\36\\tensorflow\\stream_executor\\cuda\\cuda_driver.cc:1110] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS :: No stack trace available\r\n```\r\n\r\ncuda-memcheck:\r\n```\r\n========= CUDA-MEMCHECK\r\n2018-01-02 18:15:26.061068: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\35\\tensorflow\\core\\platform\\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2\r\n2018-01-02 18:15:26.872851: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\35\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1030] Found device 0 with properties:\r\nname: GeForce GTX 770M major: 3 minor: 0 memoryClockRate(GHz): 0.797\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 3.00GiB freeMemory: 2.52GiB\r\n2018-01-02 18:15:26.873028: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\35\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 770M, pci bus id: 0000:01:00.0, compute capability: 3.0)\r\n2018-01-02 18:15:27.425673: I C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\35\\tensorflow\\core\\kernels\\cuda_solvers.cc:159] Creating CudaSolver handles for stream 00000162C1EBD150\r\n2018-01-02 18:15:30.721433: E C:\\tf_jenkins\\home\\workspace\\rel-win\\M\\windows-gpu\\PY\\35\\tensorflow\\stream_executor\\cuda\\cuda_blas.cc:551] failed to run cuBLAS routine cublasDgemv_v2: CUBLAS_STATUS_EXECUTION_FAILED\r\nTraceback (most recent call last):\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 1323, in _do_call\r\n    return fn(*args)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 1302, in _run_fn\r\n    status, run_metadata)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py\", line 473, in __exit__\r\n    c_api.TF_GetCode(self.status.status))\r\ntensorflow.python.framework.errors_impl.InternalError: Blas GEMV launch failed:  m=1, n=1\r\n         [[Node: gradients/Cholesky_grad/MatMul_1 = MatMul[T=DT_DOUBLE, transpose_a=true, transpose_b=false, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](gradients/Cholesky_grad/MatrixTriangularSolve, gradients/Cholesky_grad/MatrixBandPart)]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"test2.py\", line 11, in <module>\r\n    print(sess.run(tf.gradients(f, x), {x:np.array(1.).reshape(1, 1)}))\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 889, in run\r\n    run_metadata_ptr)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 1120, in _run\r\n    feed_dict_tensor, options, run_metadata)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 1317, in _do_run\r\n    options, run_metadata)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\client\\session.py\", line 1336, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.InternalError: Blas GEMV launch failed:  m=1, n=1\r\n         [[Node: gradients/Cholesky_grad/MatMul_1 = MatMul[T=DT_DOUBLE, transpose_a=true, transpose_b=false, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](gradients/Cholesky_grad/MatrixTriangularSolve, gradients/Cholesky_grad/MatrixBandPart)]]\r\n\r\nCaused by op 'gradients/Cholesky_grad/MatMul_1', defined at:\r\n  File \"test2.py\", line 11, in <module>\r\n    print(sess.run(tf.gradients(f, x), {x:np.array(1.).reshape(1, 1)}))\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py\", line 581, in gradients\r\n    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py\", line 353, in _MaybeCompile\r\n    return grad_fn()  # Exit early\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\gradients_impl.py\", line 581, in <lambda>\r\n    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\linalg_grad.py\", line 77, in _CholeskyGrad\r\n    math_ops.matmul(l_inverse, middle, adjoint_a=True), l_inverse)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\", line 1891, in matmul\r\n    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py\", line 2436, in _mat_mul\r\n    name=name)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2956, in create_op\r\n    op_def=op_def)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1470, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\n...which was originally created as op 'Cholesky', defined at:\r\n  File \"test2.py\", line 9, in <module>\r\n    f = tf.reduce_sum(tf.cholesky(x))\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\ops\\gen_linalg_ops.py\", line 419, in cholesky\r\n    \"Cholesky\", input=input, name=name)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2956, in create_op\r\n    op_def=op_def)\r\n  File \"C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1470, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\nInternalError (see above for traceback): Blas GEMV launch failed:  m=1, n=1\r\n         [[Node: gradients/Cholesky_grad/MatMul_1 = MatMul[T=DT_DOUBLE, transpose_a=true, transpose_b=false, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](gradients/Cholesky_grad/MatrixTriangularSolve, gradients/Cholesky_grad/MatrixBandPart)]]\r\n\r\n========= Invalid __global__ read of size 1\r\n=========     at 0x00000398 in void Eigen::internal::FullReductionKernel<int=256, int=128, Eigen::TensorEvaluator<Eigen::TensorReductionOp<Eigen::internal::SumReducer<double>, Eigen::DimensionList<__int64, unsigned __int64=2> const , Eigen::TensorGeneratorOp<tensorflow::generator::OverwriteDiagGenerator<double>, Eigen::TensorMap<Eigen::Tensor<double const , int=2, int=1, __int64>, int=16, Eigen::MakePointer> const > const , Eigen::MakePointer> const , Eigen::GpuDevice>, Eigen::internal::SumReducer<double>, __int64>(Eigen::internal::SumReducer<double>, double, __int64, Eigen::internal::SumReducer<double>::CoeffReturnType*, unsigned int*)\r\n=========     by thread (0,0,0) in block (0,0,0)\r\n=========     Address 0x00000000 is out of bounds\r\n=========     Saved host backtrace up to driver entry point at kernel launch time\r\n=========     Host Frame:C:\\WINDOWS\\SYSTEM32\\nvcuda.dll (cuTexRefSetAddress + 0x1aaaf3) [0x1b8145]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd [0x35d2]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd [0x2365]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (perftools::gputools::cuda::CUDATimer::Stop + 0x1cf760) [0x2202ba0]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (perftools::gputools::cuda::CUDATimer::Stop + 0x1d18f2) [0x2204d32]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (perftools::gputools::cuda::CUDATimer::Stop + 0x1ce985) [0x2201dc5]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::MatrixSetDiagOp<Eigen::GpuDevice,double>::Compute + 0x407) [0x13180d7]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::BaseGPUDevice::ComputeHelper + 0x4f4) [0x3aab84]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::BaseGPUDevice::Compute + 0x18a) [0x3aa3aa]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::NewLocalExecutor + 0xfdb) [0x2aefdb]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (?_Copy@?$_Func_impl@V?$_Binder@U_Unforced@std@@P8ExecutorState@?A0x4546b700@tensorflow@@EAAXUTaggedNode@345@_J@ZQEAV345@AEBU6345@AEA_J@std@@V?$allocator@H@2@X$$V@std@@EEBAPEAV?$_Func_base@X$$V@2@PEAX@Z + 0x78) [0x2b2548]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop + 0x3d9) [0x247839]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop + 0x560) [0x2479c0]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::WindowsFileSystem::Utf8ToWideChar + 0x175) [0x276795]\r\n=========     Host Frame:C:\\Users\\tomah\\AppData\\Local\\Programs\\Python\\Python35\\lib\\site-packages\\tensorflow\\python\\_pywrap_tensorflow_internal.pyd (tensorflow::WindowsFileSystem::Utf8ToWideChar + 0xc9) [0x2766e9]\r\n=========     Host Frame:C:\\WINDOWS\\System32\\ucrtbase.dll (iswascii + 0xa5) [0x1d885]\r\n=========     Host Frame:C:\\WINDOWS\\System32\\KERNEL32.DLL (BaseThreadInitThunk + 0x14) [0x11fe4]\r\n=========     Host Frame:C:\\WINDOWS\\SYSTEM32\\ntdll.dll (RtlUserThreadStart + 0x21) [0x6ef91]\r\n=========\r\n========= ERROR SUMMARY: 1 error\r\n```\r\n\r\n  "},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/374010676","html_url":"https://github.com/tensorflow/tensorflow/issues/15783#issuecomment-374010676","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/15783","id":374010676,"user":{"login":"tensorflowbutler","id":28546240,"avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-03-18T16:01:14Z","updated_at":"2018-03-18T16:01:14Z","author_association":"MEMBER","body":"Nagging Awaiting TensorFlower: It has been 14 days with no activity and the `awaiting tensorflower` label was assigned. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}