{"action":"created","issue":{"url":"https://api.github.com/repos/lattice/quda/issues/474","repository_url":"https://api.github.com/repos/lattice/quda","labels_url":"https://api.github.com/repos/lattice/quda/issues/474/labels{/name}","comments_url":"https://api.github.com/repos/lattice/quda/issues/474/comments","events_url":"https://api.github.com/repos/lattice/quda/issues/474/events","html_url":"https://github.com/lattice/quda/issues/474","id":157387000,"number":474,"title":"further twisted mass clover convergence issues","user":{"login":"kostrzewa","id":1325246,"avatar_url":"https://avatars.githubusercontent.com/u/1325246?v=3","gravatar_id":"","url":"https://api.github.com/users/kostrzewa","html_url":"https://github.com/kostrzewa","followers_url":"https://api.github.com/users/kostrzewa/followers","following_url":"https://api.github.com/users/kostrzewa/following{/other_user}","gists_url":"https://api.github.com/users/kostrzewa/gists{/gist_id}","starred_url":"https://api.github.com/users/kostrzewa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kostrzewa/subscriptions","organizations_url":"https://api.github.com/users/kostrzewa/orgs","repos_url":"https://api.github.com/users/kostrzewa/repos","events_url":"https://api.github.com/users/kostrzewa/events{/privacy}","received_events_url":"https://api.github.com/users/kostrzewa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":35,"created_at":"2016-05-29T13:45:15Z","updated_at":"2016-06-02T18:03:21Z","closed_at":null,"body":"I would like to request some help, if possible, with pin-pointing the origin of convergence issues that I see using twisted mass clover on the Jureca K80 nodes at Juelich Supercomputing Center. I don't see any of these issues on our local K20 GPU nodes in Bonn. I've compiled 97672d379fa4b7bb9cf55a7b9a305071d9dc76ba of the develop branch and use QUDA through the latest commit of the master branch of etmc/tmLQCD\r\n\r\nI invert a stochastic time-slice source on a twisted mass clover lattice and this goes through without problems (although the residual goes up for a number of iterations). This propagator is then used as a source for a sequential inversion.\r\n\r\nI've attached two log files, one of which shows an aborted inversion of the sequential propagator while the other one is successful (on the same configuration and using the same parameters). Completely disabling tuning seems to increase the probability of success, but leads to very suboptimal performance. (that is, when I don't provide a QUDA_RESOURCE_PATH and set QUDA_TUNE_NO, the inversions go throgh more freqently)\r\n\r\n[fail.log.zip](https://github.com/lattice/quda/files/288356/fail.log.zip)\r\n[success.log.zip](https://github.com/lattice/quda/files/288355/success.log.zip)\r\n\r\nOne point of potential importance might be the fact that the NUMA affinity does not work on Jureca, so I currently compile QUDA with this functionality disabled. In addition, JSC deploys CUDA with the Intel compiler only, which might be relevant. The specific versions are given as:\r\n\r\nCUDA/7.5.18\r\nMVAPICH2/2.2b-GDR\r\nIntel/2015.3.187-GCC-4.9.3-2.25\r\n\r\n@maddyscientist I haven't gotten around to reorganising the tmLQCD interface for QUDA to not reuse the parameter struct. This is irrelevant here, however, because the executable is called multiple times in each job and the parameter struct is thus always \"clean\"."},"comment":{"url":"https://api.github.com/repos/lattice/quda/issues/comments/223372684","html_url":"https://github.com/lattice/quda/issues/474#issuecomment-223372684","issue_url":"https://api.github.com/repos/lattice/quda/issues/474","id":223372684,"user":{"login":"mathiaswagner","id":4508207,"avatar_url":"https://avatars.githubusercontent.com/u/4508207?v=3","gravatar_id":"","url":"https://api.github.com/users/mathiaswagner","html_url":"https://github.com/mathiaswagner","followers_url":"https://api.github.com/users/mathiaswagner/followers","following_url":"https://api.github.com/users/mathiaswagner/following{/other_user}","gists_url":"https://api.github.com/users/mathiaswagner/gists{/gist_id}","starred_url":"https://api.github.com/users/mathiaswagner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mathiaswagner/subscriptions","organizations_url":"https://api.github.com/users/mathiaswagner/orgs","repos_url":"https://api.github.com/users/mathiaswagner/repos","events_url":"https://api.github.com/users/mathiaswagner/events{/privacy}","received_events_url":"https://api.github.com/users/mathiaswagner/received_events","type":"User","site_admin":false},"created_at":"2016-06-02T18:03:21Z","updated_at":"2016-06-02T18:03:21Z","body":"Not at all ?\n\nIf you tune multiple times the results may differ.\n\nIf you tune in run 1, the next runs (run 2 and later) should give the same result. Run1 may differ. With the reproducible branch run 1 should no love longer differ. Unless there is a bug in that branch.\n\nOn 02.06.2016, at 19:54, Bartosz Kostrzewa <notifications@github.com<mailto:notifications@github.com>> wrote:\n\n\n@maddyscientist<https://github.com/maddyscientist> Despite my using the reproducible branch, it seems that the inversion histories are not, in fact, reproducible. Am I doing something wrong or missing some kind of run-time parameter?\n\n-\nYou are receiving this because you were mentioned.\nReply to this email directly, view it on GitHub<https://github.com/lattice/quda/issues/474#issuecomment-223368945>, or mute the thread<https://github.com/notifications/unsubscribe/AETKL6dESrqCS43t0MnR09Qthwr0SrkHks5qHxf1gaJpZM4IpSwq>.\n\nNVIDIA GmbH, Wuerselen, Germany, Amtsgericht Aachen, HRB 8361\nManaging Director: Karen Theresa Burns\n\n-----------------------------------------------------------------------------------\nThis email message is for the sole use of the intended recipient(s) and may contain\nconfidential information.  Any unauthorized review, use, disclosure or distribution\nis prohibited.  If you are not the intended recipient, please contact the sender by\nreply email and destroy all copies of the original message.\n-----------------------------------------------------------------------------------\n"}}
{"actor":{"display_login":"mathiaswagner"}}