{"action":"created","issue":{"url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/199","repository_url":"https://api.github.com/repos/pluskid/Mocha.jl","labels_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/199/comments","events_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/199/events","html_url":"https://github.com/pluskid/Mocha.jl/issues/199","id":153555267,"number":199,"title":"Cuda exception","user":{"login":"simsim314","id":5075688,"avatar_url":"https://avatars.githubusercontent.com/u/5075688?v=3","gravatar_id":"","url":"https://api.github.com/users/simsim314","html_url":"https://github.com/simsim314","followers_url":"https://api.github.com/users/simsim314/followers","following_url":"https://api.github.com/users/simsim314/following{/other_user}","gists_url":"https://api.github.com/users/simsim314/gists{/gist_id}","starred_url":"https://api.github.com/users/simsim314/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simsim314/subscriptions","organizations_url":"https://api.github.com/users/simsim314/orgs","repos_url":"https://api.github.com/users/simsim314/repos","events_url":"https://api.github.com/users/simsim314/events{/privacy}","received_events_url":"https://api.github.com/users/simsim314/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-05-06T23:05:03Z","updated_at":"2016-05-11T02:38:13Z","closed_at":null,"body":"I'm getting this error, anytime I want to run CUDA in mocha. I've confirmed it in several combinations of cuDNN and CUDA version, (specifically cuDNN 4 + 3 and CUDA 7.0 and 7.5). Each time I checked I recompiled the kernels with VS2013 compiler, switch the correct paths in environment variables, and restarted windows (I work on windows10 julia 0.4.5 - everything is updated, mocha is compiled). \r\n\r\nI've successfully ran training with MOCHA_USE_NATIVE_EXT flag, and it seems I didn't do anything too stupid for the GPU (I enable _ENV[\"MOCHA_USE_CUDA\"] = \"true_\" **before** _using Mocha_), and the GPU setup looks like starting to work (initialization is working). But then I get an error. \r\n\r\nCUDA       enabled [DEV=0] (MOCHA_USE_CUDA environment variable detected)\r\nNative Ext disabled by default\r\nMocha configured, continue loading module...\r\nDefaultBackend = Mocha.GPUBackend\r\n07-May 13:52:21:INFO:root:Initializing CuDNN backend...\r\n07-May 13:52:22:INFO:root:CuDNN backend initialized!\r\n07-May 13:52:23:INFO:root:Constructing net MNIST-train on Mocha.GPUBackend...\r\n.\r\n.\r\n.\r\n.The net params\r\n.\r\n.\r\n.\r\nPlease submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.\r\nException: EXCEPTION_ACCESS_VIOLATION at 0x7ff984bacc0d -- unknown function (ip: 00007FF984BACC0D)\r\ncudnnAddTensor at  (unknown line)\r\nadd_tensor at C:\\Users\\SimSim314\\.julia\\v0.4\\Mocha\\src\\cuda/cudnn.jl:54\r\njlcall_add_tensor_2489 at  (unknown line)\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\nforward at C:\\Users\\SimSim314\\.julia\\v0.4\\Mocha\\src\\cuda\\layers/convolution.jl:89\r\njlcall_forward_2481 at  (unknown line)\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\nforward at C:\\Users\\SimSim314\\.julia\\v0.4\\Mocha\\src\\net.jl:148\r\njlcall_forward_2378 at  (unknown line)\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\ninit_solve at C:\\Users\\SimSim314\\.julia\\v0.4\\Mocha\\src\\solvers.jl:189\r\nsolve at C:\\Users\\SimSim314\\.julia\\v0.4\\Mocha\\src\\solvers.jl:234\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\njl_interpret_toplevel_expr at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\njl_interpret_toplevel_thunk_with at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\njl_eval_with_compiler_p at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\njl_parse_eval_all at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\njl_load_ at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\ninclude at boot.jl:261\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\ninclude_from_node1 at loading.jl:320\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\nprocess_options at client.jl:280\r\n_start at client.jl:378\r\njlcall__start_505 at  (unknown line)\r\njl_apply_generic at C:\\Users\\SimSim314\\AppData\\Local\\Julia-0.4.5\\bin\\libjulia.dll (unknown line)\r\nunknown function (ip: 00000000004018D0)\r\nunknown function (ip: 00000000004028AB)\r\nunknown function (ip: 000000000040140C)\r\nunknown function (ip: 000000000040153B)\r\nBaseThreadInitThunk at C:\\WINDOWS\\system32\\KERNEL32.DLL (unknown line)\r\nRtlUserThreadStart at C:\\WINDOWS\\SYSTEM32\\ntdll.dll (unknown line)\r\n"},"comment":{"url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/comments/218347167","html_url":"https://github.com/pluskid/Mocha.jl/issues/199#issuecomment-218347167","issue_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/199","id":218347167,"user":{"login":"davidparks21","id":964997,"avatar_url":"https://avatars.githubusercontent.com/u/964997?v=3","gravatar_id":"","url":"https://api.github.com/users/davidparks21","html_url":"https://github.com/davidparks21","followers_url":"https://api.github.com/users/davidparks21/followers","following_url":"https://api.github.com/users/davidparks21/following{/other_user}","gists_url":"https://api.github.com/users/davidparks21/gists{/gist_id}","starred_url":"https://api.github.com/users/davidparks21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidparks21/subscriptions","organizations_url":"https://api.github.com/users/davidparks21/orgs","repos_url":"https://api.github.com/users/davidparks21/repos","events_url":"https://api.github.com/users/davidparks21/events{/privacy}","received_events_url":"https://api.github.com/users/davidparks21/received_events","type":"User","site_admin":false},"created_at":"2016-05-11T02:38:13Z","updated_at":"2016-05-11T02:38:13Z","body":"It does sound like like you've hit the important points, sounds like you've run into just about every gotcha I bumped into and resolved them in a way that sounds logical enough to me.\r\n\r\nThis line still makes me a bit suspicious though.\r\n\r\nException: EXCEPTION_ACCESS_VIOLATION at 0x7ff984bacc0d -- unknown function (ip: 00007FF984BACC0D)\r\ncudnnAddTensor at (unknown line)\r\nadd_tensor at C:\\Users\\SimSim314.julia\\v0.4\\Mocha\\src\\cuda/cudnn.jl:54\r\n\r\nI did a little poking around google and found this seemingly related discussion:\r\n\r\nhttps://devtalk.nvidia.com/default/topic/863107/problem-with-cudnnaddtensor-function/\r\n\r\nIt's certainly possible I'm off base here, but my intuition leads me towards thinking that CuDNN v3 is being loaded, and I only know for sure that v4 works for me. I have no experience trying v3.  I did try v5, and that's not exactly the error I saw but it's suspiciously similar. \r\n\r\nWhat exactly is the output of running this command from the Julia REPL? I know you set the path, but this is the end-all verification that the expected dll is being loaded.\r\n\r\nLibdl.find_library([\"cudnn64_70.dll\", \"cudnn64_65.dll\", \"cudnn32_70.dll\", \"cudnn32_65.dll\", \"cudnn64_4.dll\"], [\"\"])\r\n\r\nAlso, take a look at this file: src/cuda/cudnn.jl (~line 41) and compare what I pasted above to what you have, I believe there were some recent commits that changed that line and it's possible you have a difference from me on that particular line of code. It's certainly the line to be suspicious of.\r\n\r\nIncidentally, the configuration I have working is: windows 10 workstation, Cuda 7.5 64 bit windows cuda toolkit, CuDNN v4"}}