{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/3405","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3405/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3405/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3405/events","html_url":"https://github.com/trilinos/Trilinos/issues/3405","id":357921777,"node_id":"MDU6SXNzdWUzNTc5MjE3Nzc=","number":3405,"title":"Tpetra_ASSUME_CUDA_AWARE_MPI affecting test performance on white","user":{"login":"kddevin","id":9597421,"node_id":"MDQ6VXNlcjk1OTc0MjE=","avatar_url":"https://avatars0.githubusercontent.com/u/9597421?v=4","gravatar_id":"","url":"https://api.github.com/users/kddevin","html_url":"https://github.com/kddevin","followers_url":"https://api.github.com/users/kddevin/followers","following_url":"https://api.github.com/users/kddevin/following{/other_user}","gists_url":"https://api.github.com/users/kddevin/gists{/gist_id}","starred_url":"https://api.github.com/users/kddevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kddevin/subscriptions","organizations_url":"https://api.github.com/users/kddevin/orgs","repos_url":"https://api.github.com/users/kddevin/repos","events_url":"https://api.github.com/users/kddevin/events{/privacy}","received_events_url":"https://api.github.com/users/kddevin/received_events","type":"User","site_admin":false},"labels":[{"id":289606028,"node_id":"MDU6TGFiZWwyODk2MDYwMjg=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/Tpetra","name":"Tpetra","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"kddevin","id":9597421,"node_id":"MDQ6VXNlcjk1OTc0MjE=","avatar_url":"https://avatars0.githubusercontent.com/u/9597421?v=4","gravatar_id":"","url":"https://api.github.com/users/kddevin","html_url":"https://github.com/kddevin","followers_url":"https://api.github.com/users/kddevin/followers","following_url":"https://api.github.com/users/kddevin/following{/other_user}","gists_url":"https://api.github.com/users/kddevin/gists{/gist_id}","starred_url":"https://api.github.com/users/kddevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kddevin/subscriptions","organizations_url":"https://api.github.com/users/kddevin/orgs","repos_url":"https://api.github.com/users/kddevin/repos","events_url":"https://api.github.com/users/kddevin/events{/privacy}","received_events_url":"https://api.github.com/users/kddevin/received_events","type":"User","site_admin":false},"assignees":[{"login":"kddevin","id":9597421,"node_id":"MDQ6VXNlcjk1OTc0MjE=","avatar_url":"https://avatars0.githubusercontent.com/u/9597421?v=4","gravatar_id":"","url":"https://api.github.com/users/kddevin","html_url":"https://github.com/kddevin","followers_url":"https://api.github.com/users/kddevin/followers","following_url":"https://api.github.com/users/kddevin/following{/other_user}","gists_url":"https://api.github.com/users/kddevin/gists{/gist_id}","starred_url":"https://api.github.com/users/kddevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kddevin/subscriptions","organizations_url":"https://api.github.com/users/kddevin/orgs","repos_url":"https://api.github.com/users/kddevin/repos","events_url":"https://api.github.com/users/kddevin/events{/privacy}","received_events_url":"https://api.github.com/users/kddevin/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-09-07T05:10:27Z","updated_at":"2018-09-07T20:29:07Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"<!---\r\nProvide a general summary of the issue in the Title above.  If this issue\r\npertains to a particular package in Trilinos, it's worthwhile to start the\r\ntitle with \"PackageName:  \".\r\n-->\r\nUsing devpack/20180521/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88 on white, the behavior of tests differs depending on the setting of Tpetra_ASSUME_CUDA_AWARE_MPI.\r\nWith Tpetra_ASSUME_CUDA_AWARE_MPI=ON, there are segfaults in MPI_Send called from Tpetra::Distributor::doPosts.  \r\nWith Tpetra_ASSUME_CUDA_AWARE_MPI=OFF, the tests run fine.  (Note that ATDM testing uses this setting.)\r\n\r\nI don't know whether this problem is caused by OpenMPI 3 (see #3356), or a problem with Tpetra, or something that I'm doing wrong with my configure/build.  @crtrott wrote the Tpetra test script that I am using (see info below), and it worked without errors in March with now obsolete devpack/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44.\r\n\r\nI've had difficulty getting Trilinos to configure and build with devpacks using OpenMPI 2.\r\nI will continue to try to get an OpenMPI 2 version to work.  I welcome an OpenMPI 2 build script that works on white; please share if you have one.  Thanks.\r\n\r\n@trilinos/tpetra @trilinos/zoltan2 @ndellingwood @kyungjoo-kim @nmhamster @crtrott \r\n\r\n\r\n## Motivation and Context\r\n<!---\r\nHow has this expectation failure affected you?  What are you trying to\r\naccomplish?  Why do we need to address this?  What does it have to do with\r\nanything?  Providing context helps us come up with a solution that is most\r\nuseful in the real world.\r\n-->\r\nWe need Tpetra to work for Zoltan2 testing.\r\nThe segfaults from the Tpetra::Distributor's MPI_Send occur in Zoltan2 testing as well; setting Tpetra_ASSUME_CUDA_AWARE_MPI=OFF allows Zoltan2 tests to run.\r\n\r\n\r\n## Steps to Reproduce\r\n<!---\r\nProvide a link to a live example, or an unambiguous set of steps to reproduce\r\nthis issue.  Include code to reproduce, if relevant. \r\n1. Do this.\r\n1. Do that.\r\n1. Shake fist angrily at computer.\r\n-->\r\nI am running the Tpetra test script described at https://github.com/trilinos/Trilinos/wiki/Tpetra-test-script.  I changed the devpack in the script to devpack/20180521/openmpi/3.1.0/gcc/7.2.0/cuda/9.2.88, because the original devpack in the script (devpack/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44) is no longer available on white.  I compared tests with -DTpetra_ASSUME_CUDA_AWARE_MPI=ON and OFF.  \r\n\r\nOn white, in my home directory, \r\nTrilinos/Obj_white/Test_white_2018_09_06_22.34.48 has tests with Tpetra_ASSUME_CUDA_AWARE_MPI=OFF, and \r\nTrilinos/Obj_white/Test_white_2018_09_06_15.35.24 has tests with Tpetra_ASSUME_CUDA_AWARE_MPI=ON.\r\n\r\nThe usual Testing/Temporary/LastTest.log shows the segfaults for the case with Tpetra_ASSUME_CUDA_AWARE_MPI=ON, and the passing tests with it OFF.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/419556912","html_url":"https://github.com/trilinos/Trilinos/issues/3405#issuecomment-419556912","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3405","id":419556912,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTU1NjkxMg==","user":{"login":"kddevin","id":9597421,"node_id":"MDQ6VXNlcjk1OTc0MjE=","avatar_url":"https://avatars0.githubusercontent.com/u/9597421?v=4","gravatar_id":"","url":"https://api.github.com/users/kddevin","html_url":"https://github.com/kddevin","followers_url":"https://api.github.com/users/kddevin/followers","following_url":"https://api.github.com/users/kddevin/following{/other_user}","gists_url":"https://api.github.com/users/kddevin/gists{/gist_id}","starred_url":"https://api.github.com/users/kddevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kddevin/subscriptions","organizations_url":"https://api.github.com/users/kddevin/orgs","repos_url":"https://api.github.com/users/kddevin/repos","events_url":"https://api.github.com/users/kddevin/events{/privacy}","received_events_url":"https://api.github.com/users/kddevin/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T20:29:07Z","updated_at":"2018-09-07T20:29:07Z","author_association":"CONTRIBUTOR","body":"Thanks for trying this, @kyungjoo-kim \r\n\r\nWith devpack/20180308/openmpi/2.1.2/gcc/7.2.0/cuda/9.0.176, I see the same problems as @kyungjoo-kim \r\n\r\nWith the IBM compiler devpack/20180521/openmpi/2.1.2/ibm/xl/16.1.0/cuda/9.2.88\r\nI get unresolved BLAS and LAPACK functions, even though the devpack loads netlib/3.8.0/ibm/xl/16.1.0 .\r\n\r\nThus, it has been impossible to test this build with OpenMPI 2 and CUDA.\r\n\r\n@trilinos/framework @bartlettroscoe Any suggestions for an OpenMPI 2  CUDA build on white?  Thanks.\r\n"}}
{"actor":{"display_login":"kddevin"}}