{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/226","repository_url":"https://api.github.com/repos/NVIDIA/caffe","labels_url":"https://api.github.com/repos/NVIDIA/caffe/issues/226/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/caffe/issues/226/comments","events_url":"https://api.github.com/repos/NVIDIA/caffe/issues/226/events","html_url":"https://github.com/NVIDIA/caffe/issues/226","id":174244794,"number":226,"title":"error == cudaSuccess (77 vs. 0) an illegal memory access was encountered","user":{"login":"szm2015","id":16157023,"avatar_url":"https://avatars.githubusercontent.com/u/16157023?v=3","gravatar_id":"","url":"https://api.github.com/users/szm2015","html_url":"https://github.com/szm2015","followers_url":"https://api.github.com/users/szm2015/followers","following_url":"https://api.github.com/users/szm2015/following{/other_user}","gists_url":"https://api.github.com/users/szm2015/gists{/gist_id}","starred_url":"https://api.github.com/users/szm2015/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szm2015/subscriptions","organizations_url":"https://api.github.com/users/szm2015/orgs","repos_url":"https://api.github.com/users/szm2015/repos","events_url":"https://api.github.com/users/szm2015/events{/privacy}","received_events_url":"https://api.github.com/users/szm2015/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-31T11:02:06Z","updated_at":"2016-09-01T14:46:07Z","closed_at":null,"body":"Hi everyone\r\nAs the title suggests I have encountered this error while training:\r\n     error == cudaSuccess (77 vs. 0) an illegal memory access was encountered\r\n\r\nAnother issue in NVIDIA/DIGITS ( #598) exits for this error, there the problem seems to have been resolved by reinstalling digits and caffe nv, but in my case:\r\n\r\nI first installed DIGITS in the normal [procedure](https://github.com/NVIDIA/DIGITS/blob/digits-4.0/docs/UbuntuInstall.md#repository-access) and have been working with it for a while without significant problem,  but in order to use multi-class detection yesterday I built caffe nvidia fork from source (as suggested in #157 ), I was then able to successfully launch multi-class detectNet training but then I get this error every now and then at apparently random iterations (once after epoch 26 then before epoch 20 and so on) It's either this or the training just stops at again a random iteration,in this latter case the GPU utilization reduces from more than 80 to less than 40 (most of the time zero) percent while nvidia-smi shows that caffe is still running on GPU and memory is also occupied. \r\n\r\nmy CUDA version:\r\nPackage: cuda\r\nStatus: install ok installed\r\nPriority: optional\r\nSection: devel\r\nInstalled-Size: 8\r\nMaintainer: cudatools <cudatools@nvidia.com>\r\nArchitecture: amd64\r\nVersion: 7.5-18\r\nDepends: cuda-7-5 (= 7.5-18)\r\nDescription: CUDA meta-package\r\n Meta-package containing all the available packages required for native CUDA\r\n development. Contains the toolkit, samples, driver and documentation.\r\n\r\nmy cuDNN version:\r\n Package: libcudnn5\r\nStatus: install ok installed\r\nPriority: optional\r\nSection: multiverse/libs\r\nInstalled-Size: 59315\r\nMaintainer: cudatools <cudatools@nvidia.com>\r\nArchitecture: amd64\r\nSource: cudnn\r\nVersion: 5.1.3-1+cuda7.5\r\nDescription: cuDNN runtime libraries\r\n cuDNN runtime libraries containing primitives for deep neural networks.\r\n\r\nand my caffe version:\r\nPackage: caffe-nv\r\nStatus: install ok installed\r\nPriority: optional\r\nSection: universe/misc\r\nInstalled-Size: 120\r\nMaintainer: Caffe Maintainers <caffe-maint@googlegroups.com>\r\nArchitecture: amd64\r\nVersion: 0.15.9-1+cuda7.5\r\n\r\nI'm using a GeForce GTX 850M.\r\n\r\nNote that before installing caffe from source and all that I have had several training and these issues (illegal memory access and getting stuck at an iteration) has just happened once or twice and now it's a recurring issue at every training (either the latter or the former)\r\n\r\nIn [here](https://github.com/BVLC/caffe/issues/4169) someone suggested to replace cudnn 5 with 4 (I don't know how to do that, simply installing libcudnn4 would do it? I thought it may cause conflict so I haven't tried it yet)\r\n\r\nI greatly appreciate your help!!!\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/comments/244102041","html_url":"https://github.com/NVIDIA/caffe/issues/226#issuecomment-244102041","issue_url":"https://api.github.com/repos/NVIDIA/caffe/issues/226","id":244102041,"user":{"login":"jeremy-rutman","id":3863744,"avatar_url":"https://avatars.githubusercontent.com/u/3863744?v=3","gravatar_id":"","url":"https://api.github.com/users/jeremy-rutman","html_url":"https://github.com/jeremy-rutman","followers_url":"https://api.github.com/users/jeremy-rutman/followers","following_url":"https://api.github.com/users/jeremy-rutman/following{/other_user}","gists_url":"https://api.github.com/users/jeremy-rutman/gists{/gist_id}","starred_url":"https://api.github.com/users/jeremy-rutman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeremy-rutman/subscriptions","organizations_url":"https://api.github.com/users/jeremy-rutman/orgs","repos_url":"https://api.github.com/users/jeremy-rutman/repos","events_url":"https://api.github.com/users/jeremy-rutman/events{/privacy}","received_events_url":"https://api.github.com/users/jeremy-rutman/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T14:46:06Z","updated_at":"2016-09-01T14:46:06Z","body":"I hit this too, running [this fork](https://github.com/daijifeng001/caffe-mnc/tree/7a28ed65cab6537eaa1e684372eafbcc0da8f185) of caffe in a docker container using cudnn4, cuda 7.5 . (Host has cuda7.5 and cudnn5). "}}
{"actor":{"display_login":"jeremy-rutman"}}