{"action":"created","issue":{"url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/438","repository_url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays","labels_url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/438/labels{/name}","comments_url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/438/comments","events_url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/438/events","html_url":"https://github.com/sourceryinstitute/OpenCoarrays/issues/438","id":252088298,"node_id":"MDU6SXNzdWUyNTIwODgyOTg=","number":438,"title":"Upstream MPICH bug: oversubscribed failed image non-deterministically hangs","user":{"login":"zbeekman","id":279612,"node_id":"MDQ6VXNlcjI3OTYxMg==","avatar_url":"https://avatars3.githubusercontent.com/u/279612?v=4","gravatar_id":"","url":"https://api.github.com/users/zbeekman","html_url":"https://github.com/zbeekman","followers_url":"https://api.github.com/users/zbeekman/followers","following_url":"https://api.github.com/users/zbeekman/following{/other_user}","gists_url":"https://api.github.com/users/zbeekman/gists{/gist_id}","starred_url":"https://api.github.com/users/zbeekman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zbeekman/subscriptions","organizations_url":"https://api.github.com/users/zbeekman/orgs","repos_url":"https://api.github.com/users/zbeekman/repos","events_url":"https://api.github.com/users/zbeekman/events{/privacy}","received_events_url":"https://api.github.com/users/zbeekman/received_events","type":"User","site_admin":false},"labels":[{"id":202472792,"node_id":"MDU6TGFiZWwyMDI0NzI3OTI=","url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/labels/bug","name":"bug","color":"fc2929","default":true},{"id":299124869,"node_id":"MDU6TGFiZWwyOTkxMjQ4Njk=","url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/labels/confirmed","name":"confirmed","color":"fbca04","default":false},{"id":301769179,"node_id":"MDU6TGFiZWwzMDE3NjkxNzk=","url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/labels/upstream-mpi-bug","name":"upstream-mpi-bug","color":"207de5","default":false},{"id":299110510,"node_id":"MDU6TGFiZWwyOTkxMTA1MTA=","url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/labels/waiting","name":"waiting","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":{"login":"afanfa","id":8441559,"node_id":"MDQ6VXNlcjg0NDE1NTk=","avatar_url":"https://avatars3.githubusercontent.com/u/8441559?v=4","gravatar_id":"","url":"https://api.github.com/users/afanfa","html_url":"https://github.com/afanfa","followers_url":"https://api.github.com/users/afanfa/followers","following_url":"https://api.github.com/users/afanfa/following{/other_user}","gists_url":"https://api.github.com/users/afanfa/gists{/gist_id}","starred_url":"https://api.github.com/users/afanfa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/afanfa/subscriptions","organizations_url":"https://api.github.com/users/afanfa/orgs","repos_url":"https://api.github.com/users/afanfa/repos","events_url":"https://api.github.com/users/afanfa/events{/privacy}","received_events_url":"https://api.github.com/users/afanfa/received_events","type":"User","site_admin":false},"assignees":[{"login":"afanfa","id":8441559,"node_id":"MDQ6VXNlcjg0NDE1NTk=","avatar_url":"https://avatars3.githubusercontent.com/u/8441559?v=4","gravatar_id":"","url":"https://api.github.com/users/afanfa","html_url":"https://github.com/afanfa","followers_url":"https://api.github.com/users/afanfa/followers","following_url":"https://api.github.com/users/afanfa/following{/other_user}","gists_url":"https://api.github.com/users/afanfa/gists{/gist_id}","starred_url":"https://api.github.com/users/afanfa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/afanfa/subscriptions","organizations_url":"https://api.github.com/users/afanfa/orgs","repos_url":"https://api.github.com/users/afanfa/repos","events_url":"https://api.github.com/users/afanfa/events{/privacy}","received_events_url":"https://api.github.com/users/afanfa/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2017-08-22T21:01:39Z","updated_at":"2018-07-01T01:49:32Z","closed_at":null,"author_association":"MEMBER","body":"<!-- Please fill out the issue template included below, failure to do -->\r\n<!-- so may result in immediate closure of your issue. -->\r\n\r\n<!-- Fill out all portions of this template that apply. Please delete -->\r\n<!-- any unnecessary sections. -->\r\n\r\n<!-- PRO TIP! Submit the pull request *before* you check any -->\r\n<!-- checkboxes. Then, use the gui/web interface to check the -->\r\n<!-- checkboxes! -->\r\n\r\n[issue stat img]: https://img.shields.io/issuestats/i/github/sourceryinstitute/OpenCoarrays.svg?style=flat-square\r\n\r\n| Avg response time              |\r\n|:------------------------------:|\r\n| ![Issue Stats][issue stat img] |\r\n\r\n# Defect/Bug Report #\r\n\r\n<!-- delete this section if it does not apply -->\r\n\r\n<!-- THE FOLLOWING INFORMATION IS MANDATORY if you would like our help -->\r\n\r\n- OpenCoarrays Version: master (f7a5f2ebeaf935a67184a978bc40177e4399b82b)\r\n- Fortran Compiler: GCC trunk & 7.2\r\n- C compiler used for building lib: Same as above\r\n- Installation method: Developer Makefiles and CMake\r\n- Output of `uname -a`: Linux & macOS\r\n- MPI library being used: MPICH 3.2, __no observed problems with OpenMPI+ULFM, private branch__\r\n- Machine architecture and number of physical cores: x86_64 8 cores\r\n- Version of CMake: 3.9\r\n\r\n## Observed Behavior ##\r\n\r\nOversubscribed fault tolerant/failed image tests hang non-deterministically when using MPICH 3.2\r\n\r\nAlso, some deprecated MPI calls create warnings, so this issue is still assigned to @afanfa to remind us to patch this.\r\n\r\n## Expected Behavior ##\r\n\r\nDon't hang, proper failed img behavior, don't use deprecated MPI calls\r\n\r\n## Steps to Reproduce ##\r\n\r\nOversubscribe failed image tests. (@afanfa & @rouson, please elaborate if you can.)\r\n\r\n\r\n[links]:#\r\n[Google]: https://www.google.com\r\n[Stack Overflow]: https://stackoverflow.com\r\n[fenced codeblocks]: https://help.github.com/articles/creating-and-highlighting-code-blocks/\r\n[GFortran bug tracker]: https://gcc.gnu.org/bugzilla/\r\n[Intel MPI]: https://software.intel.com/en-us/intel-mpi-library\r\n[MPT]: http://www.sgi.com/products/software/sps.html\r\n[MVAPICH]: http://mvapich.cse.ohio-state.edu\r\n[MPICH]: https://www.mpich.org\r\n[Open-MPI]: https://www.open-mpi.org\r\n[CONTRIBUTING.md]: https://github.com/sourceryinstitute/OpenCoarrays/blob/master/CONTRIBUTING.md\r\n[Helping Out]: https://github.com/sourceryinstitute/OpenCoarrays/blob/master/CONTRIBUTING.md#helping-out\r\n[official release]: https://github.com/sourceryinstitute/OpenCoarrays/releases\r\n[CMake install]: https://github.com/sourceryinstitute/OpenCoarrays/blob/master/INSTALL.md#cmake-scripts\r\n[homebrew]: https://brew.sh\r\n[issues page]: https://github.com/sourceryinstitute/OpenCoarrays/issues\r\n[mailing list]: https://groups.google.com/forum/#!forum/opencoarrays\r\n[install.sh]: https://github.com/sourceryinstitute/OpenCoarrays/blob/master/install.sh\r\n"},"comment":{"url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/comments/401577521","html_url":"https://github.com/sourceryinstitute/OpenCoarrays/issues/438#issuecomment-401577521","issue_url":"https://api.github.com/repos/sourceryinstitute/OpenCoarrays/issues/438","id":401577521,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTU3NzUyMQ==","user":{"login":"zbeekman","id":279612,"node_id":"MDQ6VXNlcjI3OTYxMg==","avatar_url":"https://avatars3.githubusercontent.com/u/279612?v=4","gravatar_id":"","url":"https://api.github.com/users/zbeekman","html_url":"https://github.com/zbeekman","followers_url":"https://api.github.com/users/zbeekman/followers","following_url":"https://api.github.com/users/zbeekman/following{/other_user}","gists_url":"https://api.github.com/users/zbeekman/gists{/gist_id}","starred_url":"https://api.github.com/users/zbeekman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zbeekman/subscriptions","organizations_url":"https://api.github.com/users/zbeekman/orgs","repos_url":"https://api.github.com/users/zbeekman/repos","events_url":"https://api.github.com/users/zbeekman/events{/privacy}","received_events_url":"https://api.github.com/users/zbeekman/received_events","type":"User","site_admin":false},"created_at":"2018-07-01T01:49:32Z","updated_at":"2018-07-01T01:49:32Z","author_association":"MEMBER","body":"`image_fail_and_sync_test_2` is also effected. More breadcrumbs:\r\n\r\n```\r\n      Test #69: image_fail_and_sync_test_2 .......................................................***Timeout 200.00 sec\r\n2/4: _gfortran_caf_image_status(7892) Image status of image #1 is: 0\r\n2/4: _gfortran_caf_image_status(7892) Image status of image #2 is: 0\r\n2/4: _gfortran_caf_image_status(7892) Image status of image #3 is: 0\r\n2/4: _gfortran_caf_image_status(7892) Image status of image #4 is: 0\r\n3/4: _gfortran_caf_image_status(7892) Image status of image #1 is: 0\r\n4/4: _gfortran_caf_image_status(7892) Image status of image #1 is: 0\r\n2/4: _gfortran_caf_sync_all(1537) Entering sync all.\r\n1/4: _gfortran_caf_image_status(7892) Image status of image #1 is: 0\r\n1/4: _gfortran_caf_image_status(7892) Image status of image #2 is: 0\r\n1/4: _gfortran_caf_image_status(7892) Image status of image #3 is: 0\r\n3/4: _gfortran_caf_image_status(7892) Image status of image #2 is: 0\r\n4/4: _gfortran_caf_image_status(7892) Image status of image #2 is: 0\r\n1/4: _gfortran_caf_image_status(7892) Image status of image #4 is: 0\r\n1/4: _gfortran_caf_sync_all(1537) Entering sync all.\r\n4/4: _gfortran_caf_image_status(7892) Image status of image #3 is: 0\r\n3/4: _gfortran_caf_image_status(7892) Image status of image #3 is: 0\r\n3/4: _gfortran_caf_image_status(7892) Image status of image #4 is: 0\r\n4/4: _gfortran_caf_image_status(7892) Image status of image #4 is: 0\r\n4/4: _gfortran_caf_sync_all(1537) Entering sync all.\r\n3/4: _gfortran_caf_sync_all(1537) Entering sync all.\r\n1/4: _gfortran_caf_sync_all(1553) MPI_Barrier = 0.\r\n1/4: _gfortran_caf_sync_all(1586) Leaving sync all.\r\n1/4: sync_images_internal(6780) Entering\r\n2/4: _gfortran_caf_sync_all(1553) MPI_Barrier = 0.\r\n2/4: _gfortran_caf_sync_all(1586) Leaving sync all.\r\nIMAGE FAILED!\r\n3/4: _gfortran_caf_sync_all(1553) MPI_Barrier = 0.\r\n4/4: _gfortran_caf_sync_all(1553) MPI_Barrier = 0.\r\n4/4: _gfortran_caf_sync_all(1586) Leaving sync all.\r\n4/4: sync_images_internal(6780) Entering\r\n3/4: _gfortran_caf_sync_all(1586) Leaving sync all.\r\n3/4: sync_images_internal(6780) Entering\r\n3/4: sync_images_internal(6895) Image failed, provoking error handling.\r\n3/4: sync_images_internal(6912) Leaving\r\n3/4: finalize_internal(930) (status_code = 0)\r\n3/4: failed_stopped_errorhandler_function(499) (error = 101)\r\n1/4: sync_images_internal(6895) Image failed, provoking error handling.\r\n1/4: sync_images_internal(6912) Leaving\r\n Test passed.\r\n1/4: finalize_internal(930) (status_code = 0)\r\n1/4: failed_stopped_errorhandler_function(499) (error = 101)\r\n4/4: sync_images_internal(6895) Image failed, provoking error handling.\r\n4/4: sync_images_internal(6912) Leaving\r\n4/4: finalize_internal(930) (status_code = 0)\r\n4/4: failed_stopped_errorhandler_function(499) (error = 101)\r\n4/4: failed_stopped_errorhandler_function(505) 1 images failed.\r\n1/4: failed_stopped_errorhandler_function(505) 1 images failed.\r\n1/4: failed_stopped_errorhandler_function(589) Before shrink. \r\n3/4: failed_stopped_errorhandler_function(505) 1 images failed.\r\n3/4: failed_stopped_errorhandler_function(589) Before shrink. \r\n4/4: failed_stopped_errorhandler_function(589) Before shrink. \r\n4/4: failed_stopped_errorhandler_function(591) After shrink, rc = 0.\r\n4/4: failed_stopped_errorhandler_function(599) After getting ranks, ns = 3, srank = 2, crank = 3.\r\n1/4: failed_stopped_errorhandler_function(591) After shrink, rc = 0.\r\n3/4: failed_stopped_errorhandler_function(591) After shrink, rc = 0.\r\n3/4: failed_stopped_errorhandler_function(599) After getting ranks, ns = 3, srank = 1, crank = 2.\r\n1/4: failed_stopped_errorhandler_function(599) After getting ranks, ns = 3, srank = 0, crank = 0.\r\n1/4: failed_stopped_errorhandler_function(605) After split, rc = 0, rank = 0.\r\n4/4: failed_stopped_errorhandler_function(605) After split, rc = 0, rank = 2.\r\n3/4: failed_stopped_errorhandler_function(605) After split, rc = 0, rank = 1.\r\n3/4: failed_stopped_errorhandler_function(611) After agree, flag = 0.\r\n3/4: failed_stopped_errorhandler_function(614) After rank, drank = 1.\r\n4/4: failed_stopped_errorhandler_function(611) After agree, flag = 0.\r\n3/4: failed_stopped_errorhandler_function(630) Comm_compare(*comm, CAF_COMM_WORLD, res = 0) = 0.\r\n3/4: failed_stopped_errorhandler_function(633) Comm_compare(*comm, alive_comm, res = 1) = 0.\r\n3/4: failed_stopped_errorhandler_function(637) detached win img_status.\r\n3/4: failed_stopped_errorhandler_function(638) MPI-error: err_class=50 ierr=1009407282 [Wrong synchronization of RMA calls , error stack:\r\nMPI_Win_free(135).: MPI_Win_free(win=0xe1b550) failed\r\nMPID_Win_free(160): Wrong synchronization of RMA calls ]3/4: failed_stopped_errorhandler_function(639) freed win img_status.\r\n4/4: failed_stopped_errorhandler_function(614) After rank, drank = 2.\r\n4/4: failed_stopped_errorhandler_function(630) Comm_compare(*comm, CAF_COMM_WORLD, res = 0) = 0.\r\n4/4: failed_stopped_errorhandler_function(633) Comm_compare(*comm, alive_comm, res = 1) = 0.\r\n4/4: failed_stopped_errorhandler_function(637) detached win img_status.\r\n4/4: failed_stopped_errorhandler_function(638) MPI-error: err_class=50 ierr=405427506 [Wrong synchronization of RMA calls , error stack:\r\nMPI_Win_free(135).: MPI_Win_free(win=0x144b590) failed\r\nMPID_Win_free(160): Wrong synchronization of RMA calls ]4/4: failed_stopped_errorhandler_function(639) freed win img_status.\r\n1/4: failed_stopped_errorhandler_function(611) After agree, flag = 0.\r\n1/4: failed_stopped_errorhandler_function(614) After rank, drank = 0.\r\n1/4: failed_stopped_errorhandler_function(630) Comm_compare(*comm, CAF_COMM_WORLD, res = 0) = 0.\r\n1/4: failed_stopped_errorhandler_function(633) Comm_compare(*comm, alive_comm, res = 1) = 0.\r\n1/4: failed_stopped_errorhandler_function(637) detached win img_status.\r\n1/4: failed_stopped_errorhandler_function(638) MPI-error: err_class=50 ierr=608326962 [Wrong synchronization of RMA calls , error stack:\r\nMPI_Win_free(135).: MPI_Win_free(win=0x262d590) failed\r\nMPID_Win_free(160): Wrong synchronization of RMA calls ]1/4: failed_stopped_errorhandler_function(639) freed win img_status.\r\n3/4: failed_stopped_errorhandler_function(642) (re-)created win img_status.\r\n3/4: failed_stopped_errorhandler_function(644) Win_lock_all on img_status.\r\n3/4: finalize_internal(959) MPI-error: err_class=113 ierr=6001 [Unknown error class, error stack:\r\n(unknown)(): Value of position must be nonnegative]4/4: failed_stopped_errorhandler_function(642) (re-)created win img_status.\r\n3/4: finalize_internal(959) MPI-error: err_class=5 ierr=540666373 [Invalid communicator, error stack:\r\nMPI_Send(174): MPI_Send(buf=0x6273b0, count=1, MPI_INT, dest=3, tag=424242, comm=0x84000002) failed\r\nMPI_Send(96).: Invalid communicator]3/4: finalize_internal(970) MPI-error: err_class=5 ierr=541682437 [Invalid communicator, error stack:\r\nPMPI_Errhandler_set(119): MPI_Errhandler_set(comm=0x84000002, errh=0x54000001) failed\r\nPMPI_Errhandler_set(88).: Invalid communicator]3/4: finalize_internal(974) Before MPI_Barrier(CAF_COMM_WORLD)\r\n3/4: finalize_internal(975) MPI-error: err_class=5 ierr=72953861 [Invalid communicator, error stack:\r\nPMPI_Barrier(425): MPI_Barrier(comm=0x84000002) failed\r\nPMPI_Barrier(401): Invalid communicator]3/4: finalize_internal(976) After MPI_Barrier(CAF_COMM_WORLD) = 72953861\r\n3/4: finalize_internal(1016) Freed all slave tokens.\r\n4/4: failed_stopped_errorhandler_function(644) Win_lock_all on img_status.\r\n4/4: finalize_internal(959) MPI-error: err_class=113 ierr=6001 [Unknown error class, error stack:\r\n(unknown)(): Value of position must be nonnegative]4/4: finalize_internal(959) MPI-error: err_class=5 ierr=473557509 [Invalid communicator, error stack:\r\nMPI_Send(174): MPI_Send(buf=0x6273b0, count=1, MPI_INT, dest=2, tag=424242, comm=0x84000002) failed\r\nMPI_Send(96).: Invalid communicator]4/4: finalize_internal(970) MPI-error: err_class=5 ierr=541682437 [Invalid communicator, error stack:\r\nPMPI_Errhandler_set(119): MPI_Errhandler_set(comm=0x84000002, errh=0x54000001) failed\r\nPMPI_Errhandler_set(88).: Invalid communicator]4/4: finalize_internal(974) Before MPI_Barrier(CAF_COMM_WORLD)\r\n4/4: finalize_internal(975) MPI-error: err_class=5 ierr=72953861 [Invalid communicator, error stack:\r\nPMPI_Barrier(425): MPI_Barrier(comm=0x84000002) failed\r\nPMPI_Barrier(401): Invalid communicator]4/4: finalize_internal(976) After MPI_Barrier(CAF_COMM_WORLD) = 72953861\r\n4/4: finalize_internal(1016) Freed all slave tokens.\r\n1/4: failed_stopped_errorhandler_function(642) (re-)created win img_status.\r\n1/4: failed_stopped_errorhandler_function(644) Win_lock_all on img_status.\r\n1/4: finalize_internal(959) MPI-error: err_class=113 ierr=6001 [Unknown error class, error stack:\r\n(unknown)(): Value of position must be nonnegative]1/4: finalize_internal(974) Before MPI_Barrier(CAF_COMM_WORLD)\r\n```"}}
{"actor":{"display_login":"zbeekman"}}