{"action":"created","issue":{"url":"https://api.github.com/repos/vincentlab/PyFR/issues/103","repository_url":"https://api.github.com/repos/vincentlab/PyFR","labels_url":"https://api.github.com/repos/vincentlab/PyFR/issues/103/labels{/name}","comments_url":"https://api.github.com/repos/vincentlab/PyFR/issues/103/comments","events_url":"https://api.github.com/repos/vincentlab/PyFR/issues/103/events","html_url":"https://github.com/vincentlab/PyFR/pull/103","id":151294625,"number":103,"title":"Cleanups and CUDA-aware-MPI Support","user":{"login":"FreddieWitherden","id":528611,"avatar_url":"https://avatars.githubusercontent.com/u/528611?v=3","gravatar_id":"","url":"https://api.github.com/users/FreddieWitherden","html_url":"https://github.com/FreddieWitherden","followers_url":"https://api.github.com/users/FreddieWitherden/followers","following_url":"https://api.github.com/users/FreddieWitherden/following{/other_user}","gists_url":"https://api.github.com/users/FreddieWitherden/gists{/gist_id}","starred_url":"https://api.github.com/users/FreddieWitherden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FreddieWitherden/subscriptions","organizations_url":"https://api.github.com/users/FreddieWitherden/orgs","repos_url":"https://api.github.com/users/FreddieWitherden/repos","events_url":"https://api.github.com/users/FreddieWitherden/events{/privacy}","received_events_url":"https://api.github.com/users/FreddieWitherden/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-27T04:07:14Z","updated_at":"2016-06-23T22:50:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/vincentlab/PyFR/pulls/103","html_url":"https://github.com/vincentlab/PyFR/pull/103","diff_url":"https://github.com/vincentlab/PyFR/pull/103.diff","patch_url":"https://github.com/vincentlab/PyFR/pull/103.patch"},"body":"This request cleans up various parts of PyFR, including adding support for mpi4py v2.0.  The main new feature is the ability for PyFR to exploit the fact that many MPI libraries are now CUDA aware.  This permits the CUDA backend to directly pass device pointers to various MPI routines and have the MPI library determine the most efficient means of transferring the data.  This may include the use of GPUDirect and GPUDirect over RDMA where available.\r\n\r\nSupport is enabled by setting\r\n```\r\n[backend-cuda]\r\nmpi-type = cuda-aware\r\n```\r\nwhich must of course be used with a CUDA aware MPI library (OpenMPI, MVAPICH, MVAPICH-GDR).\r\n\r\nPerformance depends heavily on the problem type, the topology of the system, and the MPI library.  As such it may not always yield an improvement."},"comment":{"url":"https://api.github.com/repos/vincentlab/PyFR/issues/comments/228209181","html_url":"https://github.com/vincentlab/PyFR/pull/103#issuecomment-228209181","issue_url":"https://api.github.com/repos/vincentlab/PyFR/issues/103","id":228209181,"user":{"login":"FreddieWitherden","id":528611,"avatar_url":"https://avatars.githubusercontent.com/u/528611?v=3","gravatar_id":"","url":"https://api.github.com/users/FreddieWitherden","html_url":"https://github.com/FreddieWitherden","followers_url":"https://api.github.com/users/FreddieWitherden/followers","following_url":"https://api.github.com/users/FreddieWitherden/following{/other_user}","gists_url":"https://api.github.com/users/FreddieWitherden/gists{/gist_id}","starred_url":"https://api.github.com/users/FreddieWitherden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FreddieWitherden/subscriptions","organizations_url":"https://api.github.com/users/FreddieWitherden/orgs","repos_url":"https://api.github.com/users/FreddieWitherden/repos","events_url":"https://api.github.com/users/FreddieWitherden/events{/privacy}","received_events_url":"https://api.github.com/users/FreddieWitherden/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T22:50:34Z","updated_at":"2016-06-23T22:50:34Z","body":"As a means of improving the start up time of PyFR on large clusters we now directly implement the ``__contains__`` method inside of ``NativeReader``.  This method is called when one executes a statement along the lines of ``name in file``.\r\n\r\nThe default implementation of this method provided by the base class works by attempting to get the item in question.  If this is successful then ``True`` is returned otherwise, if an ``AttributeError`` is raised, it is caught and ``False`` is returned.  From the standpoint of PyFR this means that statements along the lines of ``name in file`` end up reading the relevant array/record from disk should it exist.\r\n\r\nThis is a problem as the rank allocation code executes a large number of statements along these lines in order to ensure that if a mesh contains a data set ``con_pxpy`` that it also contains ``con_pypx``.  With the default implementation of ``__contains__`` this means that the root rank (which is the one performing the rank allocation) ends up reading all of the MPI interfaces.  This can result in a lot of disk I/O for a sufficiently large grid.\r\n\r\nThe new implementation avoids this reading and just asks ``h5py`` directly if the record in question exists in the file."}}
{"actor":{"display_login":"FreddieWitherden"}}