{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/6164","repository_url":"https://api.github.com/repos/BVLC/caffe","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/6164/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/6164/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/6164/events","html_url":"https://github.com/BVLC/caffe/issues/6164","id":287701977,"number":6164,"title":"BatchReindexLayer fails GPU gradient tests under CUDA v9.1","user":{"login":"MrMYHuang","id":12458706,"avatar_url":"https://avatars0.githubusercontent.com/u/12458706?v=4","gravatar_id":"","url":"https://api.github.com/users/MrMYHuang","html_url":"https://github.com/MrMYHuang","followers_url":"https://api.github.com/users/MrMYHuang/followers","following_url":"https://api.github.com/users/MrMYHuang/following{/other_user}","gists_url":"https://api.github.com/users/MrMYHuang/gists{/gist_id}","starred_url":"https://api.github.com/users/MrMYHuang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MrMYHuang/subscriptions","organizations_url":"https://api.github.com/users/MrMYHuang/orgs","repos_url":"https://api.github.com/users/MrMYHuang/repos","events_url":"https://api.github.com/users/MrMYHuang/events{/privacy}","received_events_url":"https://api.github.com/users/MrMYHuang/received_events","type":"User","site_admin":false},"labels":[{"id":56166608,"url":"https://api.github.com/repos/BVLC/caffe/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-01-11T08:34:29Z","updated_at":"2018-05-27T05:39:43Z","closed_at":null,"author_association":"NONE","body":"### Your system configuration\r\nOperating system: CentOS 7.4.1708\r\nCompiler:  x86_64-conda_cos6-linux-gnu-g++, gcc version 7.2.0 (crosstool-NG)\r\nGraphics card: nVIDIA GeForce GTX 1070\r\nCUDA version (if applicable): 9.1\r\nCUDNN version (if applicable): 7.0.5\r\nBLAS: openblas  0.2.20\r\nPython or MATLAB version (for pycaffe and matcaffe respectively):\r\nAnaconda 3 5.0.1 64-bit Python 3.6.4\r\n\r\n### Steps to reproduce\r\nAs shown in this issue [crosstool-NG compiled libraries compatibility problem](https://github.com/ContinuumIO/anaconda-issues/issues/8025#issuecomment-355612498), so I use Anaconda's gxx_linux-64 7.2.0 compiler to compile caffe (commit e93b5e20) on CentOS 7 with this \r\n[Makefile.config](https://gist.github.com/MrMYHuang/b173435687d06e20040ce93b35869db8) and these [Anaconda packages](https://gist.github.com/MrMYHuang/4068c24350a08600005f5028a9b19f83) (including libopenblas, leveldb, lmdb, opecv, protobuf, glog, gflags, py-boost, libboost, ...) and the following commands:\r\n```\r\nPATH=/cad/anaconda3/bin:/usr/bin make -j8\r\nPATH=/cad/anaconda3/bin:/usr/bin make -j8 test\r\nLD_LIBRARY_PATH=/usr/local/cuda/lib64 make runtest\r\n```\r\nHowever, ```make runtest``` failed at ```./build/test/test_batch_reindex_layer.testbin``` with these error messages:\r\n```\r\n[----------] 2 tests from BatchReindexLayerTest/3, where TypeParam = caffe::GPUDevice<double>\r\n[ RUN      ] BatchReindexLayerTest/3.TestForward\r\n[       OK ] BatchReindexLayerTest/3.TestForward (3 ms)\r\n[ RUN      ] BatchReindexLayerTest/3.TestGradient\r\n./include/caffe/test/test_gradient_check_util.hpp:175: Failure\r\nThe difference between computed_gradient and estimated_gradient is 0.68212591193169037, which exceeds threshold_ * scale, where\r\ncomputed_gradient evaluates to -0.68212591193169037,\r\nestimated_gradient evaluates to 0, and\r\nthreshold_ * scale evaluates to 0.01.\r\ndebug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,0; feat = -0.18315754813835761; objective+ = 3.4152213335738013; objective- = 3.4152213335738013\r\n...\r\n```\r\n\r\nAfter countless caffe compilations and tests, finally, I find a ***workaround*** to this problem: I add a line ```NVCCFLAGS += -G``` to Makefile and it changes from\r\n```\r\n...\r\n# Debugging\r\nifeq ($(DEBUG), 1)\r\n        COMMON_FLAGS += -DDEBUG -g -O0\r\n        NVCCFLAGS += -G\r\nelse\r\n        COMMON_FLAGS += -DNDEBUG -O2\r\nendif\r\n...\r\n```\r\nto\r\n```\r\n...\r\n# Debugging\r\nifeq ($(DEBUG), 1)\r\n        COMMON_FLAGS += -DDEBUG -g -O0\r\n        NVCCFLAGS += -G\r\nelse\r\n        COMMON_FLAGS += -DNDEBUG -O2\r\n        NVCCFLAGS += -G\r\nendif\r\n...\r\n```\r\nThen, compiling caffe again... and ```make runtest``` passes without failure!\r\n\r\nI think the problem could be unrelated to Python and x86_64-conda_cos6-linux-gnu-g++ compiler, but related to nvcc (CUDA 9.1). So, this problem might be reproduced with other g++ compiler!? Actually, I see someone has the [same problem](http://blog.csdn.net/pzh16789/article/details/78913107) with Ubuntu 16.04 + CUDA 9.1. I hope someone can fix this problem."},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/392306722","html_url":"https://github.com/BVLC/caffe/issues/6164#issuecomment-392306722","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/6164","id":392306722,"user":{"login":"CDLuminate","id":5723047,"avatar_url":"https://avatars3.githubusercontent.com/u/5723047?v=4","gravatar_id":"","url":"https://api.github.com/users/CDLuminate","html_url":"https://github.com/CDLuminate","followers_url":"https://api.github.com/users/CDLuminate/followers","following_url":"https://api.github.com/users/CDLuminate/following{/other_user}","gists_url":"https://api.github.com/users/CDLuminate/gists{/gist_id}","starred_url":"https://api.github.com/users/CDLuminate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CDLuminate/subscriptions","organizations_url":"https://api.github.com/users/CDLuminate/orgs","repos_url":"https://api.github.com/users/CDLuminate/repos","events_url":"https://api.github.com/users/CDLuminate/events{/privacy}","received_events_url":"https://api.github.com/users/CDLuminate/received_events","type":"User","site_admin":false},"created_at":"2018-05-27T05:39:43Z","updated_at":"2018-05-27T05:39:43Z","author_association":"CONTRIBUTOR","body":"Has anyone tested CUDA 9.2?"}}
{"actor":{"display_login":"CDLuminate"}}