{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/1530","repository_url":"https://api.github.com/repos/open-mpi/ompi","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/1530/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/1530/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/1530/events","html_url":"https://github.com/open-mpi/ompi/issues/1530","id":147002593,"number":1530,"title":"MPI Errhandler causes segfault on receipt of signal","user":{"login":"rhc54","id":2440928,"avatar_url":"https://avatars.githubusercontent.com/u/2440928?v=3","gravatar_id":"","url":"https://api.github.com/users/rhc54","html_url":"https://github.com/rhc54","followers_url":"https://api.github.com/users/rhc54/followers","following_url":"https://api.github.com/users/rhc54/following{/other_user}","gists_url":"https://api.github.com/users/rhc54/gists{/gist_id}","starred_url":"https://api.github.com/users/rhc54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhc54/subscriptions","organizations_url":"https://api.github.com/users/rhc54/orgs","repos_url":"https://api.github.com/users/rhc54/repos","events_url":"https://api.github.com/users/rhc54/events{/privacy}","received_events_url":"https://api.github.com/users/rhc54/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/open-mpi/ompi/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/open-mpi/ompi/milestones/14","html_url":"https://github.com/open-mpi/ompi/milestones/v1.10.3","labels_url":"https://api.github.com/repos/open-mpi/ompi/milestones/14/labels","id":1404927,"number":14,"title":"v1.10.3","description":"","creator":{"login":"ompiteam","id":2165735,"avatar_url":"https://avatars.githubusercontent.com/u/2165735?v=3","gravatar_id":"","url":"https://api.github.com/users/ompiteam","html_url":"https://github.com/ompiteam","followers_url":"https://api.github.com/users/ompiteam/followers","following_url":"https://api.github.com/users/ompiteam/following{/other_user}","gists_url":"https://api.github.com/users/ompiteam/gists{/gist_id}","starred_url":"https://api.github.com/users/ompiteam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ompiteam/subscriptions","organizations_url":"https://api.github.com/users/ompiteam/orgs","repos_url":"https://api.github.com/users/ompiteam/repos","events_url":"https://api.github.com/users/ompiteam/events{/privacy}","received_events_url":"https://api.github.com/users/ompiteam/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":9,"state":"open","created_at":"2015-11-10T17:01:10Z","updated_at":"2016-04-08T18:27:59Z","due_on":"2016-06-01T07:00:00Z","closed_at":null},"comments":8,"created_at":"2016-04-08T18:27:59Z","updated_at":"2016-04-13T23:55:57Z","closed_at":null,"body":"In trying to track down an apparent \"hang\" on the 1.10.3 branch, I discovered that the job wasn't actually hanging. Instead, the job was taking a long time to complete because nearly every proc was segfaulting and writing out a core dump - and it was taking so long to write out all those core dumps that MTT thought we had hung.\r\n\r\nThe test is Intel MPI_Errhandler_fatal1_f (a few others show the same symptom - this is just the one I was working with). Everything starts correctly, and then rank=0 calls MPI_Abort. ORTE correctly identifies the situation and issues \"kill\" signals to the other procs, starting with SIGCONT followed by SIGTERM, and then SIGKILL.\r\n\r\nThe output results look okay:\r\n\r\n```shell\r\n$ mpirun -n 16 ./MPI_Errhandler_fatal1_f\r\n MPITEST_INFO (         0): Starting test MPI_Errhandler_fatal     \r\n MPITEST_INFO (         0): This test should abort after printing the results\r\n MPITEST_INFO (         0): message, otherwise a f.a.i.l.u.r.e is noted\r\nMPITEST_results: MPI_Errhandler_fatal all tests PASSED (        16)\r\n[bend001:21445] *** An error occurred in MPI_Send\r\n[bend001:21445] *** reported by process [139981018365953,21474836480]\r\n[bend001:21445] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0\r\n[bend001:21445] *** MPI_ERR_RANK: invalid rank\r\n[bend001:21445] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,\r\n[bend001:21445] ***    and potentially your MPI job)\r\n```\r\n\r\nHowever, the writing out of the core files takes a long time, and the shell prompt doesn't return until that dump is completed. Thus, MTT thinks the job timed out and tries to kill it - and fails, because the mpirun is actually already dead.\r\n\r\nI have no idea why this test is segfaulting upon receipt of a signal. Can someone please investigate?\r\n\r\n@ggouaillardet @bosilca @hjelmn @jsquyres \r\n\r\nNot sure which of you might have time and knowledge to dig this one out."},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/209694991","html_url":"https://github.com/open-mpi/ompi/issues/1530#issuecomment-209694991","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/1530","id":209694991,"user":{"login":"rhc54","id":2440928,"avatar_url":"https://avatars.githubusercontent.com/u/2440928?v=3","gravatar_id":"","url":"https://api.github.com/users/rhc54","html_url":"https://github.com/rhc54","followers_url":"https://api.github.com/users/rhc54/followers","following_url":"https://api.github.com/users/rhc54/following{/other_user}","gists_url":"https://api.github.com/users/rhc54/gists{/gist_id}","starred_url":"https://api.github.com/users/rhc54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhc54/subscriptions","organizations_url":"https://api.github.com/users/rhc54/orgs","repos_url":"https://api.github.com/users/rhc54/repos","events_url":"https://api.github.com/users/rhc54/events{/privacy}","received_events_url":"https://api.github.com/users/rhc54/received_events","type":"User","site_admin":false},"created_at":"2016-04-13T23:55:57Z","updated_at":"2016-04-13T23:55:57Z","body":"I don't have psm support on my system, nor are any of those components built. I can't swear that isn't the problem - maybe some other component is opening something that can cause the problem. For example, I do have libfabric installed, and the usnic btl is opening it, so perhaps this problem is creeping in that way?"}}