{"action":"created","issue":{"url":"https://api.github.com/repos/lattice/quda/issues/482","repository_url":"https://api.github.com/repos/lattice/quda","labels_url":"https://api.github.com/repos/lattice/quda/issues/482/labels{/name}","comments_url":"https://api.github.com/repos/lattice/quda/issues/482/comments","events_url":"https://api.github.com/repos/lattice/quda/issues/482/events","html_url":"https://github.com/lattice/quda/issues/482","id":162373520,"number":482,"title":"Documentation for QUDA Dslash test","user":{"login":"dipsankarb","id":9132645,"avatar_url":"https://avatars.githubusercontent.com/u/9132645?v=3","gravatar_id":"","url":"https://api.github.com/users/dipsankarb","html_url":"https://github.com/dipsankarb","followers_url":"https://api.github.com/users/dipsankarb/followers","following_url":"https://api.github.com/users/dipsankarb/following{/other_user}","gists_url":"https://api.github.com/users/dipsankarb/gists{/gist_id}","starred_url":"https://api.github.com/users/dipsankarb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dipsankarb/subscriptions","organizations_url":"https://api.github.com/users/dipsankarb/orgs","repos_url":"https://api.github.com/users/dipsankarb/repos","events_url":"https://api.github.com/users/dipsankarb/events{/privacy}","received_events_url":"https://api.github.com/users/dipsankarb/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-06-27T04:48:13Z","updated_at":"2016-06-27T17:41:34Z","closed_at":"2016-06-27T16:44:20Z","body":"Hi,\r\n\r\nI am trying to run QUDA Dslash test with MPI. When I run with 2 processes (on one node with multiple GPUs) I get this error: Running from test directory inside my build.\r\n\r\n$ mpirun -np 2 gpu01 gpu01 ./dslash_test\r\nERROR: ERROR: Communication grid size declared via initCommsGridQuda() does not match total number of MPI ranks (1 != 2)Communication grid size declared via initCommsGridQuda() does not match total number of MPI ranks (1 != 2) (rank 0, host gpu01.cluster, /home/n01/quda/lib/comm_mpi.cpp:68 in comm_init())\r\n (rank 1, host gpu01.cluster, /home/n01/quda/lib/comm_mpi.cpp:68 in comm_init())\r\n       last kernel called was (name=,volume=,aux=)\r\n       last kernel called was (name=,volume=,aux=)\r\n[cli_1]: [cli_0]: aborting job:\r\napplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 0\r\naborting job:\r\napplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 1\r\n\r\nI have enabled support for multi gpu and GDR while configuring. I think I need to pass some additional parameters to dslash for running it properly with multiple processes. Unfortunately I am unable to locate any documentation for QUDA. The one pointed here : http://lattice.github.io/quda-ref/v0.8.0/\r\ndoes not exist. Please help."},"comment":{"url":"https://api.github.com/repos/lattice/quda/issues/comments/228818920","html_url":"https://github.com/lattice/quda/issues/482#issuecomment-228818920","issue_url":"https://api.github.com/repos/lattice/quda/issues/482","id":228818920,"user":{"login":"dipsankarb","id":9132645,"avatar_url":"https://avatars.githubusercontent.com/u/9132645?v=3","gravatar_id":"","url":"https://api.github.com/users/dipsankarb","html_url":"https://github.com/dipsankarb","followers_url":"https://api.github.com/users/dipsankarb/followers","following_url":"https://api.github.com/users/dipsankarb/following{/other_user}","gists_url":"https://api.github.com/users/dipsankarb/gists{/gist_id}","starred_url":"https://api.github.com/users/dipsankarb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dipsankarb/subscriptions","organizations_url":"https://api.github.com/users/dipsankarb/orgs","repos_url":"https://api.github.com/users/dipsankarb/repos","events_url":"https://api.github.com/users/dipsankarb/events{/privacy}","received_events_url":"https://api.github.com/users/dipsankarb/received_events","type":"User","site_admin":false},"created_at":"2016-06-27T17:41:34Z","updated_at":"2016-06-27T17:41:34Z","body":"I am not seeing good scaling with the increase in number of GPUs. For example, if I run keeping xdim, ydim, zdim and tdim fixed at 48. Then I run as follows:\r\n\r\n$ mpirun -np 2 gpu01 gpu01 ./dslash_test  --xdim 48 --ydim 48 --zdim 48 --tdim 48 --xgridsize 2 --ygridsize 1\r\n\r\nNumbers:\r\n\r\nxgrid 2 ygrid 1 (2 procs on 1 K80 1 node which is basically 2 GPUs)\r\n\r\nendQuda Total time = 0.1826 secs\r\ninitQuda-endQuda Total time = 93.1645 secs\r\nQUDA Total time = 3.44325 secs\r\n\r\nxgrid 2 ygrid 2 (4 procs 1 node 2 K80s)\r\n\r\nendQuda Total time = 0.352839 secs\r\ninitQuda-endQuda Total time = 131.197 secs\r\nQUDA Total time = 29.6026 secs\r\n\r\nxgrid 4 ygrid 2 (8 procs 2 nodes 4 K80s (2 per node))\r\n\r\nendQuda Total time = 0.280718 secs\r\ninitQuda-endQuda Total time = 156.035 secs\r\nQUDA Total time = 35.4498 secs\r\n\r\nIn this context, please let me know if the numbers are making sense. Am I running it correctly\r\nas its supposed to for best performance?\r\n\r\nI am using a CUDA aware\r\nMPI and GDR benefits will come into picture only if the communications are CUDA aware i.e; \r\nthe MPI comms are working directly over GPU buffers and are not staged through the host. I can\r\ngo into the code and see what is implemented but from initial runs I can see that the communications\r\nare happening over host which does not help with GDR. Please let me know you thoughts/comments\r\non this.\r\n\r\nAlso, the documentation in the github wiki is very brief and is not very informative. Will help if I can\r\nget a bit more informative wiki/documentation. If the one that is supposed to be in the broken link is\r\nback up, I can go through it and see if it helps.\r\n\r\nThanks again."}}
{"actor":{"display_login":"dipsankarb"}}