{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/3844","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3844/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3844/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3844/events","html_url":"https://github.com/trilinos/Trilinos/issues/3844","id":379332727,"node_id":"MDU6SXNzdWUzNzkzMzI3Mjc=","number":3844,"title":"ATDM_CONFIG_MPI_EXEC Causes Problems on Chama","user":{"login":"jmgate","id":20327215,"node_id":"MDQ6VXNlcjIwMzI3MjE1","avatar_url":"https://avatars3.githubusercontent.com/u/20327215?v=4","gravatar_id":"","url":"https://api.github.com/users/jmgate","html_url":"https://github.com/jmgate","followers_url":"https://api.github.com/users/jmgate/followers","following_url":"https://api.github.com/users/jmgate/following{/other_user}","gists_url":"https://api.github.com/users/jmgate/gists{/gist_id}","starred_url":"https://api.github.com/users/jmgate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmgate/subscriptions","organizations_url":"https://api.github.com/users/jmgate/orgs","repos_url":"https://api.github.com/users/jmgate/repos","events_url":"https://api.github.com/users/jmgate/events{/privacy}","received_events_url":"https://api.github.com/users/jmgate/received_events","type":"User","site_admin":false},"labels":[{"id":816763194,"node_id":"MDU6TGFiZWw4MTY3NjMxOTQ=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM","name":"ATDM","color":"ea88a5","default":false},{"id":933882543,"node_id":"MDU6TGFiZWw5MzM4ODI1NDM=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM%20Config","name":"ATDM Config","color":"006b75","default":false},{"id":1106096209,"node_id":"MDU6TGFiZWwxMTA2MDk2MjA5","url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM%20DevOps","name":"ATDM DevOps","color":"bfdadc","default":false},{"id":287520108,"node_id":"MDU6TGFiZWwyODc1MjAxMDg=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":{"login":"bartlettroscoe","id":1959736,"node_id":"MDQ6VXNlcjE5NTk3MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"assignees":[{"login":"bartlettroscoe","id":1959736,"node_id":"MDQ6VXNlcjE5NTk3MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2018-11-09T21:45:15Z","updated_at":"2018-12-13T20:48:58Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Hey @bartlettroscoe, I'm having trouble getting a build of EMPIRE up and running on `chama`.  It'll build, but I'm seeing all of EMPIRE-PIC's tests fail (see [EMPIRE_EM-Plasma-Trilinos-chama-test-all-of-EMPIRE-intel-opt-openmp](https://jenkins-srn.sandia.gov/user/jmgate/my-views/view/EMPIRE/job/EMPIRE_EM-Plasma-Trilinos-chama-test-all-of-EMPIRE-intel-opt-openmp/9/consoleFull)).  I think what's happening is on `chama` the `ATDM_CONFIG_MPI_EXEC` variable gets set to `srun` in [cmake/std/atdm/common/toss3/environment.sh](https://github.com/trilinos/Trilinos/blob/7e0342c8d2ab8855fe5112b483284a8837911d20/cmake/std/atdm/common/toss3/environment.sh#L74).  When we test EMPIRE, we wind up submitting the `ctest` command via `srun`.  Unfortunately each EMPIRE-PIC regression test winds up running\r\n```\r\n$Trilinos_MPI_EXEC -n $NUM_PROCS $EXE --i=$INPUT\r\n```\r\n\r\nwhere `$TRILINOS_MPI_EXEC` is `$ATDM_CONFIG_MPI_EXEC`, which is `srun`, so we're effectively invoking `srun` within something that's been submitted via `srun`.  I don't know slurm inside and out, but that sounds like a problem.\r\n\r\nThe question, then, is what should we do about it?  In the CMake file that's used for all of EMPIRE-PIC's regression tests, do we need to replace `$Trilinos_MPI_EXEC` with `$(which mpiexec)` or something to that effect?  Or do we need to do something more clever with the `$ATDM_CONFIG_MPI_EXEC` variable in `cmake/std/atdm/common/toss3/environment.sh` such that it works for whatever you need it to do, but then it doesn't also mess up EMPIRE's testing on `toss3` systems?\r\n\r\n@bathmatt, did you set up our CMake stuff to use `$Trilinos_MPI_EXEC`?  Do you know the rationale behind it?"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/447115658","html_url":"https://github.com/trilinos/Trilinos/issues/3844#issuecomment-447115658","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3844","id":447115658,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NzExNTY1OA==","user":{"login":"bartlettroscoe","id":1959736,"node_id":"MDQ6VXNlcjE5NTk3MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2018-12-13T20:48:58Z","updated_at":"2018-12-13T20:48:58Z","author_association":"MEMBER","body":"> I haven't had time to get back to this. We wound up subdividing the task and the part that triggered this problem has yet to be tackled.\r\n\r\n@jmgate, okay, then we can leave this open for now.  Let me know when you can try the approach used by the ATDM Trilinos ctest -S drivers documented in the script [atdm/chama/local-driver.sh](https://github.com/trilinos/Trilinos/blob/develop/cmake/ctest/drivers/atdm/chama/local-driver.sh) using the [atdm_run_script_on_compute_node](https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/common/toss3/environment.sh#L111) bash function."}}
{"actor":{"display_login":"bartlettroscoe"}}