{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/3753","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/3753/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/3753/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/3753/events","html_url":"https://github.com/tensorflow/models/issues/3753","id":308649222,"node_id":"MDU6SXNzdWUzMDg2NDkyMjI=","number":3753,"title":"[deeplab] CUDA_ERROR_LAUNCH_FAILED while using demo script","user":{"login":"ogruhlke94","id":18286022,"node_id":"MDQ6VXNlcjE4Mjg2MDIy","avatar_url":"https://avatars3.githubusercontent.com/u/18286022?v=4","gravatar_id":"","url":"https://api.github.com/users/ogruhlke94","html_url":"https://github.com/ogruhlke94","followers_url":"https://api.github.com/users/ogruhlke94/followers","following_url":"https://api.github.com/users/ogruhlke94/following{/other_user}","gists_url":"https://api.github.com/users/ogruhlke94/gists{/gist_id}","starred_url":"https://api.github.com/users/ogruhlke94/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ogruhlke94/subscriptions","organizations_url":"https://api.github.com/users/ogruhlke94/orgs","repos_url":"https://api.github.com/users/ogruhlke94/repos","events_url":"https://api.github.com/users/ogruhlke94/events{/privacy}","received_events_url":"https://api.github.com/users/ogruhlke94/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-03-26T16:42:58Z","updated_at":"2018-11-01T13:02:57Z","closed_at":"2018-11-01T13:02:57Z","author_association":"NONE","body":"### System information\r\n- **What is the top-level directory of the model you are using**:\r\ndeeplab\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\nNo\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:\r\nUbuntu 16.04\r\n- **TensorFlow installed from (source or binary)**:\r\nsource\r\n- **TensorFlow version (use command below)**:\r\n1.6\r\n- **Bazel version (if compiling from source)**:\r\n0.11.1\r\n- **CUDA/cuDNN version**:\r\nCuda 9.1 / cuDNN 7.1.2\r\n- **GPU model and memory**:\r\nGeForce GTX TITAN  - 6GB\r\n- **Exact command to reproduce**:\r\n\r\n### Describe the problem\r\nI'm trying to execute the provided demo script _local_test.sh_. I only made some changes from other issues regarding to python3 and changed the batch_size to 1 due to my limited VRAM. Also I increased the iteration number to 1000.\r\nAfter a different number of iterations (between 10 and 200) a CUDA_ERROR_LAUNCH_FAILED occurs  (see provided log).\r\nI also tried using the binary version of tensorflow with Cuda 9.0 and cuDNN 7.0. My current nvidia driver version is 390.30.\r\nI watched the GPU memory usage with nvidia-smi and it was always about 300MB free VRAM - so a out of memory issue seems unlikely.\r\n\r\n### Source code / logs\r\n```\r\n2018-03-26 18:22:08.100135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties: \r\nname: GeForce GTX TITAN major: 3 minor: 5 memoryClockRate(GHz): 0.8755\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 5.93GiB freeMemory: 5.83GiB\r\n2018-03-26 18:22:08.100148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0\r\n2018-03-26 18:22:08.268167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5627 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN, pci bus id: 0000:01:00.0, compute capability: 3.5)\r\nINFO:tensorflow:Restoring parameters from /home/ogruhlke/Schreibtisch/tensorflow_models/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train/model.ckpt-0\r\nINFO:tensorflow:Running local_init_op.\r\nINFO:tensorflow:Done running local_init_op.\r\nINFO:tensorflow:Starting Session.\r\nINFO:tensorflow:Saving checkpoint to path /home/ogruhlke/Schreibtisch/tensorflow_models/models/research/deeplab/datasets/pascal_voc_seg/exp/train_on_trainval_set/train/model.ckpt\r\nINFO:tensorflow:Starting Queues.\r\nINFO:tensorflow:global_step/sec: 0\r\nINFO:tensorflow:Recording summary at step 0.\r\nINFO:tensorflow:global step 10: loss = 3.2638 (0.176 sec/step)\r\nINFO:tensorflow:global step 20: loss = 0.9162 (0.174 sec/step)\r\nINFO:tensorflow:global step 30: loss = 0.8637 (0.174 sec/step)\r\n2018-03-26 18:22:29.385213: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED\r\n2018-03-26 18:22:29.385252: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1\r\nAborted (core dumped)\r\n```"}}
{"actor":{"display_login":"wt-huang"}}