{"action":"created","issue":{"url":"https://api.github.com/repos/lattice/quda/issues/522","repository_url":"https://api.github.com/repos/lattice/quda","labels_url":"https://api.github.com/repos/lattice/quda/issues/522/labels{/name}","comments_url":"https://api.github.com/repos/lattice/quda/issues/522/comments","events_url":"https://api.github.com/repos/lattice/quda/issues/522/events","html_url":"https://github.com/lattice/quda/issues/522","id":186061435,"number":522,"title":"Heavy memory burden on reduce.cu?","user":{"login":"cpviolator","id":2357719,"avatar_url":"https://avatars.githubusercontent.com/u/2357719?v=3","gravatar_id":"","url":"https://api.github.com/users/cpviolator","html_url":"https://github.com/cpviolator","followers_url":"https://api.github.com/users/cpviolator/followers","following_url":"https://api.github.com/users/cpviolator/following{/other_user}","gists_url":"https://api.github.com/users/cpviolator/gists{/gist_id}","starred_url":"https://api.github.com/users/cpviolator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpviolator/subscriptions","organizations_url":"https://api.github.com/users/cpviolator/orgs","repos_url":"https://api.github.com/users/cpviolator/repos","events_url":"https://api.github.com/users/cpviolator/events{/privacy}","received_events_url":"https://api.github.com/users/cpviolator/received_events","type":"User","site_admin":false},"labels":[{"id":74179,"url":"https://api.github.com/repos/lattice/quda/labels/question","name":"question","color":"4f6af0","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-29T05:44:03Z","updated_at":"2016-10-30T23:51:01Z","closed_at":null,"body":"I'm posting this as a question rather that a bug because it's an issue related to a particular system (Stampede) that has arisen in the past few weeks.\r\n\r\nI was working on the development brach pulled on the 22nd of September and I had no issues with compilation. I recently gave instructions to colleagues about how to add additions to QUDA, and instructed them to simply pull the most recent development commit and add the changes. However, I found that when I tested this myself, QUDA threw the error:\r\n\r\n`std::bad_alloc()`\r\n\r\nduring compilation, indicating that the make process had ran out of memory. The offending kernel is reduce_quda.o. I just want to ask if this kernel has recently (since 22nd Sep) had any revisions that would inflate the memory burden. Like I say, this issue is related only to Stampede as I can successfully compile the more recent QUDA commits other systems (nvscocial.) I even tried making with just one core to ensure all available memory was given to one job at a time, but this fails too.\r\n\r\nOn Stampede, the version I pulled from 22nd of September compiles OK, with and without the additions. Recent versions, with and without additions, fail to compile throwing the above error. Right now I'm compiling QUDA commits one by one to see where the compilation fails in order to pin the specific commit, and I'll report back with findings. Any clues would be appreciated."},"comment":{"url":"https://api.github.com/repos/lattice/quda/issues/comments/257189487","html_url":"https://github.com/lattice/quda/issues/522#issuecomment-257189487","issue_url":"https://api.github.com/repos/lattice/quda/issues/522","id":257189487,"user":{"login":"cpviolator","id":2357719,"avatar_url":"https://avatars.githubusercontent.com/u/2357719?v=3","gravatar_id":"","url":"https://api.github.com/users/cpviolator","html_url":"https://github.com/cpviolator","followers_url":"https://api.github.com/users/cpviolator/followers","following_url":"https://api.github.com/users/cpviolator/following{/other_user}","gists_url":"https://api.github.com/users/cpviolator/gists{/gist_id}","starred_url":"https://api.github.com/users/cpviolator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpviolator/subscriptions","organizations_url":"https://api.github.com/users/cpviolator/orgs","repos_url":"https://api.github.com/users/cpviolator/repos","events_url":"https://api.github.com/users/cpviolator/events{/privacy}","received_events_url":"https://api.github.com/users/cpviolator/received_events","type":"User","site_admin":false},"created_at":"2016-10-30T23:51:01Z","updated_at":"2016-10-30T23:51:01Z","body":"On Stampede I use CUDA 7.5, gcc 4.9.3 and mvapich 2.1. Here's a rundown of the builds I tried using a core to make, with the results:\r\n\r\n```\r\nhowarth@Stampede-/work/03653/howarth/MEM_BUG$ tail -n 6 quda-*/build/log*\r\n==> quda-06da5f9c8206e908376c8b381329c2b67c3e2a95/build/loggy.log <==\r\n  /work/03653/howarth/MEM_BUG/quda-06da5f9c8206e908376c8b381329c2b67c3e2a95/build/lib/CMakeFiles/quda.dir//./quda_generated_reduce_quda.cu.o\r\n\r\n\r\nmake[2]: *** [lib/CMakeFiles/quda.dir/quda_generated_reduce_quda.cu.o] Error 1\r\nmake[1]: *** [lib/CMakeFiles/quda.dir/all] Error 2\r\nmake: *** [all] Error 2\r\n\r\n==> quda-127153033a85cf03f8b7f5f9785933807c662282/build/loggy.log <==\r\nLinking CXX executable staggered_invert_test\r\n[ 98%] Built target staggered_invert_test\r\nScanning dependencies of target su3_test\r\n[100%] Building CXX object tests/CMakeFiles/su3_test.dir/su3_test.cpp.o\r\nLinking CXX executable su3_test\r\n[100%] Built target su3_test\r\n\r\n==> quda-24cf95ec3d00585285d214ece4c22f585933b674/build/loggy.log <==\r\nLinking CXX executable staggered_invert_test\r\n[ 98%] Built target staggered_invert_test\r\nScanning dependencies of target su3_test\r\n[100%] Building CXX object tests/CMakeFiles/su3_test.dir/su3_test.cpp.o\r\nLinking CXX executable su3_test\r\n[100%] Built target su3_test\r\n\r\n==> quda-582e8b2131c5a5df030237cd7c2485b2a05b8ea1/build/loggy.log <==\r\n  /work/03653/howarth/MEM_BUG/quda-582e8b2131c5a5df030237cd7c2485b2a05b8ea1/build/lib/CMakeFiles/quda.dir//./quda_generated_reduce_quda.cu.o\r\n\r\n\r\nmake[2]: *** [lib/CMakeFiles/quda.dir/quda_generated_reduce_quda.cu.o] Error 1\r\nmake[1]: *** [lib/CMakeFiles/quda.dir/all] Error 2\r\nmake: *** [all] Error 2\r\n\r\n==> quda-6f324e66ccda19c43bca1af2ab3d2cc07ea2674f/build/loggy.log <==\r\nLinking CXX executable staggered_invert_test\r\n[ 98%] Built target staggered_invert_test\r\nScanning dependencies of target su3_test\r\n[100%] Building CXX object tests/CMakeFiles/su3_test.dir/su3_test.cpp.o\r\nLinking CXX executable su3_test\r\n[100%] Built target su3_test\r\n\r\n==> quda-ad2114c2c2459c73be901d2c6deac80a0505f437/build/loggy.log <==\r\n  /work/03653/howarth/MEM_BUG/quda-ad2114c2c2459c73be901d2c6deac80a0505f437/build/lib/CMakeFiles/quda.dir//./quda_generated_reduce_quda.cu.o\r\n\r\n\r\nmake[2]: *** [lib/CMakeFiles/quda.dir/quda_generated_reduce_quda.cu.o] Error 1\r\nmake[1]: *** [lib/CMakeFiles/quda.dir/all] Error 2\r\nmake: *** [all] Error 2\r\n```\r\nAll the builds that fail seem to fail on reduce_quda.o Here are the dates of commits:\r\n\r\n`6f324e6` Sep 20th: success\r\n`ad2114c2` Sep 22nd: fail\r\n`06da5f9` Sep 23rd: fail\r\n`24cf95e` Sep 26th: success\r\n`1271530` Sep 27th: success\r\n`582e8b2` Oct 17th: fail\r\n\r\nSo it's a bit of a mixed bag. Not sure what to make of it. I guess I can use a working build on Stampede and perhaps try to replicate these results at a later date. If the build results differ, it's probably just an issue with the available memory on the login nodes. Though I have found that the latest commit `582e8b2` has consistently failed on Stampede."}}
{"actor":{"display_login":"cpviolator"}}