{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/1264","repository_url":"https://api.github.com/repos/NVIDIA/DIGITS","labels_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/1264/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/1264/comments","events_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/1264/events","html_url":"https://github.com/NVIDIA/DIGITS/issues/1264","id":188968998,"number":1264,"title":"CURAND_STATUS_LAUNCH_FAILURE using new Titan X Pascal","user":{"login":"NeuroRoboticTech","id":4786881,"avatar_url":"https://avatars.githubusercontent.com/u/4786881?v=3","gravatar_id":"","url":"https://api.github.com/users/NeuroRoboticTech","html_url":"https://github.com/NeuroRoboticTech","followers_url":"https://api.github.com/users/NeuroRoboticTech/followers","following_url":"https://api.github.com/users/NeuroRoboticTech/following{/other_user}","gists_url":"https://api.github.com/users/NeuroRoboticTech/gists{/gist_id}","starred_url":"https://api.github.com/users/NeuroRoboticTech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeuroRoboticTech/subscriptions","organizations_url":"https://api.github.com/users/NeuroRoboticTech/orgs","repos_url":"https://api.github.com/users/NeuroRoboticTech/repos","events_url":"https://api.github.com/users/NeuroRoboticTech/events{/privacy}","received_events_url":"https://api.github.com/users/NeuroRoboticTech/received_events","type":"User","site_admin":false},"labels":[{"id":191002061,"url":"https://api.github.com/repos/NVIDIA/DIGITS/labels/caffe","name":"caffe","color":"755638","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-11-13T12:59:54Z","updated_at":"2016-11-16T11:34:57Z","closed_at":null,"body":"I have been using Digits with caffe for a while now with a GeForce GTX 960. I wanted some more power so I shelled out a bunch of money for a new Titan X Pascal GPU. I installed it in the secondary PCI-e slot so I could use the GTX primarily for graphics, and use the Titan for GPU. I can run deviceQuery and see both GPUs. I have also run the bandwidth test on the new Titan and it passed, so the GPU is working. Initially I was using digits version 4.1-dev, and nvidia/caffe 0.15.9, and I was able to figure out how to change the digits config file to set it up so I could use both of them. However, when I clone any of my previous jobs that ran fine on the GTX and try and run them on the Titan I get an error. \r\n\r\n```\r\nI1112 06:19:01.239487  5120 solver.cpp:362] Iteration 0, Testing net (#0)\r\nI1112 06:19:04.057446  5120 blocking_queue.cpp:50] Data layer prefetch queue empty\r\nI1112 06:23:06.287995  5120 solver.cpp:429]     Test net output #0: accuracy = 0.0070898\r\nI1112 06:23:06.288074  5120 solver.cpp:429]     Test net output #1: loss = 3.04448 (* 1 = 3.04448 loss)\r\nF1112 06:23:06.447123  5120 math_functions.cu:396] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE\r\n*** Check failure stack trace: ***\r\n@     0x7fa1f5014daa  (unknown)\r\n@     0x7fa1f5014ce4  (unknown)\r\n@     0x7fa1f50146e6  (unknown)\r\n@     0x7fa1f5017687  (unknown)\r\n@     0x7fa1f5737cd4  (unknown)\r\n@     0x7fa1f5767f55  (unknown)\r\n@     0x7fa1f56d87d8  (unknown)\r\n@     0x7fa1f56d8b57  (unknown)\r\n@     0x7fa1f57116fc  (unknown)\r\n@     0x7fa1f5711fce  (unknown)\r\n@           0x40af36  (unknown)\r\n@           0x40867c  (unknown)\r\n@     0x7fa1f3b17f45  (unknown)\r\n@           0x408e4d  (unknown)\r\n@              (nil)  (unknown)\r\n```\r\n\r\nThis is a very simple test network I am using just to see if things are running correctly. All jobs I have tried to run on the Titan eventually fail this way after churning for a while, but If I clone it and run it again on the GTX it runs perfectly fine. While doing some research I found a link on this issues forum that seemed to imply that I might need CUDA 8. Since I was on CUDA 7.5 I decided to move up to 8. I had to upgrade pretty much everything. I switched over to CUDA 8 and upgraded to cuDNN 5.1, and then I had to rebuild the latest opencv with CUDA 8. I pulled the latest nvidia/caffe and rebuilt and installed it. Since Digits 5.0 was just released with semantic segmentation I pulled that from github and used it instead of the 4.1-dev I was using. However, I have the exact same problem. I can run semantic segmentation tasks on digits 5.0 just fine with my older GTX card, but when I clone that job and try and run it on the new Titan X it fails with a CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE. \r\n\r\nI am unsure if this is a dgits or caffe error or cuda or what, and I am stumped. Does anyone have any ideas on why this is not running on my new, and very expensive GPU card, but runs fine on the older, cheaper one? Does anyone have any suggestions on how I can get some more info on why it is failing? I have attached the caffe log.\r\n\r\n[caffe_output.log.zip](https://github.com/NVIDIA/DIGITS/files/587751/caffe_output.log.zip)\r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/comments/260924295","html_url":"https://github.com/NVIDIA/DIGITS/issues/1264#issuecomment-260924295","issue_url":"https://api.github.com/repos/NVIDIA/DIGITS/issues/1264","id":260924295,"user":{"login":"gheinrich","id":3889770,"avatar_url":"https://avatars.githubusercontent.com/u/3889770?v=3","gravatar_id":"","url":"https://api.github.com/users/gheinrich","html_url":"https://github.com/gheinrich","followers_url":"https://api.github.com/users/gheinrich/followers","following_url":"https://api.github.com/users/gheinrich/following{/other_user}","gists_url":"https://api.github.com/users/gheinrich/gists{/gist_id}","starred_url":"https://api.github.com/users/gheinrich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gheinrich/subscriptions","organizations_url":"https://api.github.com/users/gheinrich/orgs","repos_url":"https://api.github.com/users/gheinrich/repos","events_url":"https://api.github.com/users/gheinrich/events{/privacy}","received_events_url":"https://api.github.com/users/gheinrich/received_events","type":"User","site_admin":false},"created_at":"2016-11-16T11:34:57Z","updated_at":"2016-11-16T11:34:57Z","body":"You can find examples on the https://github.com/shelhamer/fcn.berkeleyvision.org repository.\r\n\r\nAre you still using nv-caffe 0.15.9? I recall there were some issues on this version. Can you try to upgrade to [0.15.13](https://github.com/NVIDIA/caffe/releases/tag/v0.15.13)?"}}
{"actor":{"display_login":"gheinrich"}}