{"action":"created","issue":{"url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/1322","labels_url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/1322/labels{/name}","comments_url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/1322/comments","events_url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/1322/events","html_url":"https://github.com/ComputationalRadiationPhysics/picongpu/issues/1322","id":125387971,"number":1322,"title":"Freeze of picongpu if used MPI_Allgather in plugin","user":{"login":"theZiz","id":385145,"avatar_url":"https://avatars.githubusercontent.com/u/385145?v=3","gravatar_id":"","url":"https://api.github.com/users/theZiz","html_url":"https://github.com/theZiz","followers_url":"https://api.github.com/users/theZiz/followers","following_url":"https://api.github.com/users/theZiz/following{/other_user}","gists_url":"https://api.github.com/users/theZiz/gists{/gist_id}","starred_url":"https://api.github.com/users/theZiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/theZiz/subscriptions","organizations_url":"https://api.github.com/users/theZiz/orgs","repos_url":"https://api.github.com/users/theZiz/repos","events_url":"https://api.github.com/users/theZiz/events{/privacy}","received_events_url":"https://api.github.com/users/theZiz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/labels/plugin","name":"plugin","color":"cf6800"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/milestones/2","html_url":"https://github.com/ComputationalRadiationPhysics/picongpu/milestones/Open%20Beta","labels_url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/milestones/2/labels","id":395839,"number":2,"title":"Open Beta","description":"Tasks to complete until the release of our open beta (version 0.1)\r\n\r\nThe [beta](https://en.wikipedia.org/wiki/Software_release_life_cycle#Beta) release shall be useful for *physicist* that used a Particle-in-Cell code *before*. Experience with *compiling libraries from source* and a *good Linux knowledge* is required for the initial installation.\r\n","creator":{"login":"ax3l","id":1353258,"avatar_url":"https://avatars.githubusercontent.com/u/1353258?v=3","gravatar_id":"","url":"https://api.github.com/users/ax3l","html_url":"https://github.com/ax3l","followers_url":"https://api.github.com/users/ax3l/followers","following_url":"https://api.github.com/users/ax3l/following{/other_user}","gists_url":"https://api.github.com/users/ax3l/gists{/gist_id}","starred_url":"https://api.github.com/users/ax3l/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ax3l/subscriptions","organizations_url":"https://api.github.com/users/ax3l/orgs","repos_url":"https://api.github.com/users/ax3l/repos","events_url":"https://api.github.com/users/ax3l/events{/privacy}","received_events_url":"https://api.github.com/users/ax3l/received_events","type":"User","site_admin":false},"open_issues":139,"closed_issues":1077,"state":"open","created_at":"2013-08-06T16:38:57Z","updated_at":"2016-01-07T13:36:04Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2016-01-07T12:48:01Z","updated_at":"2016-01-07T13:40:39Z","closed_at":null,"body":"Hi,\r\n\r\nI noticed a bug in picongpu while writing at isaac. I managed to write a minimal example plugin, which shows the error. The plugin called \"MPIStressTest\" is avalaible at my picongpu fork in the branch \"mpistresstest\": https://github.com/theZiz/picongpu/tree/mpistresstest\r\n\r\nThe plugin uses this code in the `notify()` call:\r\n```C++\r\ntestClass.update( cellDescription );\r\n//Some sample MPI magic\r\nint numProc;\r\nMPI_Comm_size(mpi_world, &numProc);\r\nfloat buffer[numProc];\r\nfloat my_value = float(rand());\r\nif (mode & 1)\r\n    MPI_Allgather( &my_value, 1, MPI_FLOAT, buffer, 1, MPI_FLOAT, mpi_world);\r\nif (mode & 2)\r\n    MPI_Bcast( buffer, numProc, MPI_FLOAT, 0, mpi_world);\r\n```\r\nThe `mode` in line 7 and 9 is a parameter set in the submit cfg. The applications freezes if the mode is 1 or 3, so if \"MPI_Allgather\" is called. However it may be, that MPI_Bcast makes picongpu freeze, too, from time to time. I was just not able to repdroduce this.\r\n\r\nHowever this MPI call is the reason for the freeze, but the application does not freeze at this point. It freezes in the `update` method of the dummy test class `testClass` with this implementation:\r\n```\r\nvoid update( MappingDesc *cellDescription )\r\n{\r\n\tconst SubGrid<simDim>& subGrid = Environment< simDim >::get().SubGrid();\r\n\tDataConnector &dc = Environment< simDim >::get().DataConnector(); \r\n\tFieldE * pField = &(dc.getData< FieldE > (FieldE::getName(), true));\r\n\tDataSpace< simDim > guarding = SuperCellSize::toRT() * cellDescription->getGuardingSuperCells();\r\n\t//printf(\"get device data box\\n\");\r\n\tFieldE::DataBoxType dataBox = pField->getDeviceDataBox();\r\n\t//printf(\"done\\n\");\r\n\tFieldE::DataBoxType shifted = dataBox.shift( guarding );\r\n}\r\n```\r\nThe function `FieldE::DataBoxType dataBox = pField->getDeviceDataBox();` does not return (Checked with the surrounded printfs) for one node. However every else node will block at any later MPI call, too, because of the frozen rank.\r\n\r\nI defined DEBUG_EVENTS in src/libPMacc/include/eventSystem/Manager.tpp to have some debug informations at this point. If the appliaction freezes this is printed to stdout:\r\n```\r\n[1,13]<stdout>:TaskReceive 1 2\r\n[1,13]<stdout>:TaskReceiveMPI 1 2\r\n ```\r\nMost of the time it happens right at the start, but sometimes after more than 100 steps, sometimes is runs the full simulation without problems. Most of the time the error is exactly as posted, but sometimes the 2 is switched for 3 or the message appears twice, but in this case the ranks in front of <stdout> are different.\r\n\r\nI used the Laserwakefield example with default parameters. In my submit cfg I have only the test plugin activated. I am using 16 gpus with x=2, y=4, z=2 and a grid of 128 x 512 x 128. If I set x=1, y=16 and z=1 the bug appears more rarely, but still occurs.\r\n\r\nIt doesn't matter, how I start the job (direct qsub or qsub with bash).\r\n\r\nIf you need more informations about my test settings, don't hesitate to ask!\r\n\r\nGreetings, Ziz"},"comment":{"url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/comments/169666828","html_url":"https://github.com/ComputationalRadiationPhysics/picongpu/issues/1322#issuecomment-169666828","issue_url":"https://api.github.com/repos/ComputationalRadiationPhysics/picongpu/issues/1322","id":169666828,"user":{"login":"ax3l","id":1353258,"avatar_url":"https://avatars.githubusercontent.com/u/1353258?v=3","gravatar_id":"","url":"https://api.github.com/users/ax3l","html_url":"https://github.com/ax3l","followers_url":"https://api.github.com/users/ax3l/followers","following_url":"https://api.github.com/users/ax3l/following{/other_user}","gists_url":"https://api.github.com/users/ax3l/gists{/gist_id}","starred_url":"https://api.github.com/users/ax3l/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ax3l/subscriptions","organizations_url":"https://api.github.com/users/ax3l/orgs","repos_url":"https://api.github.com/users/ax3l/repos","events_url":"https://api.github.com/users/ax3l/events{/privacy}","received_events_url":"https://api.github.com/users/ax3l/received_events","type":"User","site_admin":false},"created_at":"2016-01-07T13:40:39Z","updated_at":"2016-01-07T13:40:39Z","body":"Thx for the report and minimal example, that is excellent.\r\n\r\nI think you are using `MPI_Allgather` wrong: the first argument (result) is also an array of the same size as the input in your example."}}