{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/2422","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2422/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2422/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2422/events","html_url":"https://github.com/trilinos/Trilinos/issues/2422","id":306870584,"number":2422,"title":"Define better strategy for managing threaded testing with Trilinos","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"id":816763194,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM","name":"ATDM","color":"ea88a5","default":false},{"id":289605971,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Kokkos","name":"Kokkos","color":"5319e7","default":false},{"id":289606028,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Tpetra","name":"Tpetra","color":"5319e7","default":false},{"id":287520105,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-03-20T13:42:27Z","updated_at":"2018-03-23T21:37:06Z","closed_at":null,"author_association":"MEMBER","body":"**CC:**: @trilinos/framework, @trilinos/kokkos, @trilinos/tpetra, @nmhamster, @rppawlo \r\n\r\n## Next Action Status\r\n\r\nSet up a meeting to discuss current status of threaded testing in Trilinos and some steps to try to address the issues ...\r\n\r\n## Description\r\n\r\nIt seems the testing strategy in trilinos to test with threading is to build threaded code and then run all of the threaded tests with same number of threads (such as by setting `export OMP_NUM_THREADS=2` when using OpenMP ) and then running the test suite with `ctest -j<N>` with that fixed number of threads has some issues.\r\n\r\nFirst, with some configurations and systems, running with any `<N>` with `ctest -j<N>` will result in all of the test executables binding to the same threads on the same cores making things run very slowly such as described in https://github.com/trilinos/Trilinos/issues/2398#issuecomment-374379614.\r\n\r\nSecond, even when one does not experience this problem of binding to the same cores (which is not always a problem), this approach does not make very good usage the test machine because it assumes that every MPI process is multi-threaded with Kokkos, which is not true.  So this will result in not keeping many of the cores busy running code and making it take longer to complete the tests.\r\n\r\nThird, having to run the entire Trilinos test suite with a fixed number of threads like with `export OMP_NUM_THREADS=2` or `export OMP_NUM_THREADS=4` does not result in very good testing or results in very expensive testing having to run the entire Trilinos test suite multiple times.  It has been observed that defects occur when some thread counts are used like `export OMP_NUM_THREADS=5`, for example.  This would be like having every MPI test in Trilinos run with exactly the same number of MPI processes which would not result in very good testing (which is not the case in Trilinos as several tests are run with different numbers of MPI processes).\r\n\r\n## Proposed Solutions\r\n\r\nFirst, the issue of binding multiple MPI processes to the same cores should be solved using the updated ctest property `PROCESSOR_AFFINITY` that Kitware added to ctest in a recent merge to the cmake 'master' branch (and therefore will go out in the CMake 3.12 release).  This newly updated ctest property needs to tested out with some use cases on systems where this is known to be a problem.  (Running with `mpiexec -np <N> --report-bindings <exec>` should help debug this.)\r\n\r\nSecond, we should investigate how to define some new cmake wrapper functions (e.g. which calls [TRIBITS_ADD_TEST()](see https://tribits.org/doc/TribitsDevelopersGuide.html#tribits-add-test)) that allow defining the number of threads used by each test individually, just like we do with the number of MPI processes.  For example, for simple single threaded executables, we could define a function like:\r\n\r\n```\r\n  TRILINOS_ADD_THREADED_TEST( <exeName>\r\n    [any arguments passed to TRIBITS_ADD_TEST()]\r\n    NUM_THREADS_PER_PROCESS <numThreads>\r\n    )\r\n```\r\n\r\nThat function would then set the number of threads based on the enabled threading model known at configure time.  For OpenMP, it would set the env var `OMP_NUM_THREADS=<numThreads>` and for other threading models would pass in `--kokkos-threads=<numThreads>` and would compute `NUM_TOTAL_CORES_USED <numTotalCoresUsed>` (interpreted as max number of threads) passed to the `TRIBITS_ADD_TEST()` function as `<numThreads> * <numMpiProcs>`.  If the `NUM_THREADS_PER_PROCESS <numThreads>` argument was missing, this could use a default number of threads (e.g. global configure argument `Trilinos_DEFAULT_NUM_THREADS` with default `1`).  If the computed `NUM_TOTAL_CORES_USED <numTotalCoresused>` was larger than `${MPI_EXEC_MAX_NUMPROCS}` (which should be set to the maximum number of threads that can be used run on that machine when using threading), then the test would get excluded and a message would be printed to STDOUT.  Such a CMake function should be pretty easy to write, if you know the threading model used by Kokkos at configure time.\r\n\r\nSome helper functions could also be added to define tests using [TRIBITS_ADD_ADVANCED_TEST()](https://tribits.org/doc/TribitsDevelopersGuide.html#tribits-add-advanced-test) where individual `TEST_<idx> EXEC <exeRootName>` executions could be threaded.  (A genetic interface for that might be difficult but with some helper functions constructing such a test should be easy.)\r\n\r\n## Definition of Done:\r\n\r\n* ???\r\n\r\n## Tasks:\r\n\r\n1. Reproduce the problem of binding different MPI processes to the same cores and then test the new ctest property `PROCESSOR_AFFINITY` on CMake 'master' branch to see if it fixes the problem.\r\n1. Set up a meeting to discuss current status of threaded testing in Trilinos and some steps to try to address the issues.\r\n1. Experiment with a `TRILINOS_ADD_THREADED_TEST()` function and test with different Kokkos threading models?\r\n1. ???\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/375805099","html_url":"https://github.com/trilinos/Trilinos/issues/2422#issuecomment-375805099","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2422","id":375805099,"user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2018-03-23T21:37:06Z","updated_at":"2018-03-23T21:37:06Z","author_association":"MEMBER","body":"Shoot, it looks like the problem with trying to run multiple MPI jobs at the same time and slowing each other down may also be a problem with CUDA on GPUs as well, as described in https://github.com/trilinos/Trilinos/issues/2446#issuecomment-375804502.\r\n\r\n@nmhamster, we really need to figure out how to even just manage running multiple mpi jobs on the same nodes at the same time and not have them step on each other.\r\n"}}
{"actor":{"display_login":"bartlettroscoe"}}