{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/172","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/172/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/172/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/172/events","html_url":"https://github.com/trilinos/Trilinos/issues/172","id":138310982,"number":172,"title":"Create ATTBDevEnv.cmake module to automatically use loaded env for Advanced Technology Test Bed platforms","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Framework","name":"Framework","color":"bfd4f2"},{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/trilinos/Trilinos/milestones/19","html_url":"https://github.com/trilinos/Trilinos/milestones/Advanced%20Technology%20Test%20Bed%20Issue","labels_url":"https://api.github.com/repos/trilinos/Trilinos/milestones/19/labels","id":1660388,"number":19,"title":"Advanced Technology Test Bed Issue","description":"This milestone is a catch-all for issues with Trilinos arising on the Advanced Technology Test Bed machines that don't seem to be general to other platforms.  These are important machines for several programs.","creator":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":0,"state":"open","created_at":"2016-03-22T17:03:33Z","updated_at":"2016-03-22T17:21:06Z","due_on":"2016-12-31T05:00:00Z","closed_at":null},"comments":52,"created_at":"2016-03-03T22:05:44Z","updated_at":"2016-03-23T18:48:29Z","closed_at":null,"body":"Just like we want to do for SEMS environments (see #158), we want to create a file called ATTBDevEnv.cmake (stands for Advanced Technology Test Bed Development Environment) that will define the TriBITS CMake variables to find all of the right libraries and include directories when the ATTB environment is loaded.\r\n\r\nThe basic idea is that a developer will load a single module like:\r\n\r\n```\r\n$ module load devpack/openmpi/1.10.0/gcc/4.8.4/cuda/7.5.18\r\n```\r\n\r\nand that module will load a bunch of other modules.  These modules will define a bunch of environment variables of the form `<TPLName>_ROOT` (where `<TPLName>` = `HDF5`, `METIS`, `PNETCDF`, etc.).  Under these you would find:\r\n\r\n```\r\n${<TPLName>_ROOT}/include/\r\n${<TPLName>_ROOT}/lib/\r\n```\r\n\r\n(or whereever the libraries exist).\r\n\r\nIn addition, the modules set up the variables pointing to the MPI compiler wrappers like `MPICC`, `MPICXX`, and `MPIFC`.\r\n\r\nIn addition, the top-level module would define the environment variable `ATTB_ENV` which would be set and will be non-empty when the env is loaded (later we can look at the value of `ATTB_ENV` to be able to point to different versions of the env).  The goal behind the ATTB_ENV is that we would put in automatic logic like:\r\n\r\n```\r\nIF (NOT \"${ENV{ATTB_ENV}}\" STREQUAL \"\")\r\n  INCLUDE(<some-base>/ATTBDevEnv.cmake)\r\nENDIF()\r\n```\r\n\r\nThe idea would be to have Trilinos (or even TriBITS in some smart way) run this block of code and be able to automatically pick up the dev env and set the TriBITS CMake TPL vars correctly so someone could just load the top-level module and then configure Trilinos with:\r\n\r\n```\r\ncmake -DTPL_ENABLE_MPI=ON -DTPL_ENABLE_Netcdf=ON ... \\\r\n  -DTrilinos_ENABLE_SECACAS=ON -DTrilinos_ENABLE_Panzer=ON ... \\\r\n   ... \\\r\n  ${TRILINOS_DIR}\r\n```\r\n\r\nand it would automatically know where to find the include dirs and libraries for all of the TPLs enabled (implicitly or explicitly).\r\n\r\nWhat makes this tricky is that the ATTB env is set up to only use static libraries.  But they build all of these with `-fPIC` so you should still be able to build shared libraries in Trilinos and downstream.\r\n\r\nCC list: @rppawlo, @bathmatt, @nmhamster\r\n\r\n**Next Action Status**\r\n\r\nTrilinos now automatically detects and sets up for ATTB dev env on `hansen` and `shiller`.  Testing on other machine (see below) ...\r\n\r\n**Tasks:**\r\n\r\n1. Get configure scripts for Trilinos+Drekar [Done]\r\n2. Create ATTBDevEnv.cmake that successfully configures and builds Trilinos+Drekar [Done]\r\n3. Set up Trilinos+Drekar to automatically include ATTBDevEnv.cmake on ATTB machine `hansen` [Done]\r\n4. Get ATTBDevEnv.cmake working on `shiller` [Done]\r\n5. See about setting policy https://cmake.org/cmake/help/v3.5/policy/CMP0060.html, test on Shiller and Hanssen and require CMake 3.3+ … \r\n6. Get working with `OMPI_CXX=nvcc_wrapper` on `hansen` and `shiller` ...\r\n7. Add `-march=core-avx2 ` to `CMAKE_[C|CXX|Fortran]_FLAGS` to improve vectorization ...\r\n8. Get ATTBDevEnv.cmake working on `shepard`:\r\n  - Module `devpack/openmpi/1.10.0/intel/16.1.056/cuda/none` is missing It is missing `BLAS_ROOT` and `LAPACK_ROOT` ... \r\n9. Get ATTBDevEnv.cmake working on `white` ...\r\n10. Get ATTBDevEnv.cmake working on `ride` ...\r\n11. Test full Trilinos+Drekar configure on all ATTB machines:\r\n  - `hansen`\r\n  - `shiller`\r\n  - `shepard`\r\n  - `white`\r\n  - `ride`\r\n  - ???\r\n12. Document configuration of Trilinos using ATTBDevEnv.cmake on \"Heterogeneous Advanced Architecture Platforms (HAAPs)\" wiki page ...\r\n13. Set up Nightly builds posted to Trilinos CDash on ATTB platforms and hand over builds to SEMS team to run with Jenkins ...\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/200491902","html_url":"https://github.com/trilinos/Trilinos/issues/172#issuecomment-200491902","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/172","id":200491902,"user":{"login":"nmhamster","id":8905126,"avatar_url":"https://avatars.githubusercontent.com/u/8905126?v=3","gravatar_id":"","url":"https://api.github.com/users/nmhamster","html_url":"https://github.com/nmhamster","followers_url":"https://api.github.com/users/nmhamster/followers","following_url":"https://api.github.com/users/nmhamster/following{/other_user}","gists_url":"https://api.github.com/users/nmhamster/gists{/gist_id}","starred_url":"https://api.github.com/users/nmhamster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmhamster/subscriptions","organizations_url":"https://api.github.com/users/nmhamster/orgs","repos_url":"https://api.github.com/users/nmhamster/repos","events_url":"https://api.github.com/users/nmhamster/events{/privacy}","received_events_url":"https://api.github.com/users/nmhamster/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T18:48:29Z","updated_at":"2016-03-23T18:48:29Z","body":"`libsci` should be loaded automagically by the PrgEnv-{gnu, cray, intel, pgi} selection on the Cray. Its MKL where it may be more problematic for the Cray environment (it needs to use Intel's module convention for `MKLROOT`). I'd be happy to work with you on that."}}