{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/482","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/482/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/482/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/482/events","html_url":"https://github.com/trilinos/Trilinos/issues/482","id":163500779,"number":482,"title":"Set up robust portable pre-push and post-push CI tools and process based on the SEMS Dev Env","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Framework","name":"Framework","color":"bfd4f2"},{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/trilinos/Trilinos/milestones/27","html_url":"https://github.com/trilinos/Trilinos/milestones/Improve%20productivity,%20stability,%20and%20quality%20of%20Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/milestones/27/labels","id":1776612,"number":27,"title":"Improve productivity, stability, and quality of Trilinos","description":"This is a catch-all epic for stories related to improvements in the (integrated) productivity of all Trilinos developers, the stability of the main branches, and increasing the overall quality of higher-maturity and higher-consequence Trilinos code.","creator":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":0,"state":"open","created_at":"2016-05-19T14:30:14Z","updated_at":"2016-07-02T04:07:25Z","due_on":null,"closed_at":null},"comments":19,"created_at":"2016-07-02T04:07:25Z","updated_at":"2016-10-27T13:32:00Z","closed_at":null,"body":"**Next Action Status:**\r\n\r\nCreated drafts for `load_ci_sems_dev_env.sh` and `checkin-test-sems.sh`: Next: Put together updated set of PT vs. ST packages and TPLs (see #410) and discuss topic at the next Trilinos Leaders Meeting ...\r\n\r\n**Blocked By:** #158, #410, #362\r\n\r\n**Blocking:** #380\r\n\r\n**Related To:** #370, #475, #476\r\n\r\n**CC:** @trilinos/framework \r\n\r\n**Description:**\r\n\r\nTrilinos has not had an effective pre-push CI development process for many years.  When the checkin-test.py script was first created (back in 2008 or so), the primary stack of packages was based on Epetra and the main external dependencies were C/C++/Fortran compilers and BLAS and LAPACK.  Those dependencies and the major Trilinos customers at the time were used to select the initial set of Primary Tested (initially called Primary Stable) packages that is being used to this day.  However, since that time, many new Trilinos packages have been added and important Trilinos customers are relying on many of these newer packages (e.g. SEACAS, STK, Tpetra, Phalanx, Panzer, etc.).  In addition, these new Trilinos packages require more dependencies than just BLAS and LAPACK and now TPLs like Boost, HDF5, NetCDF, ParMETIS, SuperLU and others used by Trilinos are also very important to many Trilinos customers.\r\n\r\nAnother problem with the current pre-push CI testing processes with Trilinos is that Trilinos developers have a variety of different types of machines, OSs, versions of compilers, TPL implementations, etc. that they use to develop on and push changes for Trilinos.  This has resulted in people who tried to use the checkin-test.py script to suffer failed pushes due to failing tests on their machine not triggered by their changes.  In contract, projects that have a uniform pre-push CI testing env don't experience these types of problems.  One example of such a project is CASL VERA that uses TriBITS and the checkin-test.py script and has a set of uniform development machines where developers almost never see tests that fail in their build of the code that passed on another developer's build.  Therefore, the only failed builds and tests are due to their own local changes.  In that project, there is no trepidation to running the checkin-test.py script and everyone uses it uniformly for nearly every push.\r\n\r\nAnother problem with the current CI testing process for Trilinos is that the post-push CI server that posts to CDash enables a different set of packages and TPLs from what the pre-push CI build does (and of course uses different compilers, MPI, etc.).  Therefore, a CI build/test failure seen on CDash may not be seen with the checkin-test.py script locally and visa vera.  This makes it difficult for developers to determine if the failures they are seeing on their own machine are due to their local changes or due differences with the env on their machine compared to the machine running the CI build posting to CDash, if it is due to a different set of enabled packages and TPL or something else.\r\n\r\nAs a result, the stability of the main Trilinos development branch (now the 'develop' branch, see #370) has degraded from what it was 5+ years ago.  This is a problem because Trilinos needs to have a more stable 'develop' branch in order to more frequently update from the 'develop' branch to the 'master' branch (see #370).\r\n\r\nThis story is to address all of these shortcomings of the current Trilinos CI testing process.  The new SEMS Dev Env (#158) provides an opportunity to create a fairly portable (at least for SNL staff members) uniform pre-push and post-push CI testing environment for the first time.\r\n\r\nHere is the plan for setting up a more effective CI process based on the SEMS Dev Env, the checkin-test.py script, and CTest/CDash:\r\n\r\n1. **Select a standard pre-push CI build env based on the SEMS Dev Env:**  Currently, GCC 4.7.2 and OpenMPI 1.6.5 are being used for the post-push CI build that posts to CDash.  These selections should be reexamined and potentially changed.  This will be used to create a standard `load_ci_sems_dev_env.sh` script, which just calls the `local_sems_dev_env.sh` script with the selections.\r\n\r\n2. **Select an expanded/revised set of Primary Tested (PT) packages and TPLs:** This revised set should be based on the most important packages and TPLs to current Trilinos customers.  Any important TPL not already supported by the SEMS Dev Env may need to be added (i.e. to the Trilinos space under the /projects/ NFS mount).  Revising the set of PT packages and TPLs is being addressed in #410.\r\n\r\n3. **Set up a standard checkin-test-sems.sh script that all Trilinos developers can use to push changes to the Trilinos 'develop' branch:** This should automatically load the correct standard SEMS Dev Env by sourcing `load_ci_sems_dev_env.sh`.  This should likely only run a single build of Trilinos to speed up the testing/push process.  (If there is a single build is would likely include `-DTPL_ENABLE_MPI=ON -DCMAKE_BULD_TYPE=RELEASE -DTriinos_ENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=ON -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON -DTrilinos_ENABLE_FLOAT=OFF -DTrilinos_ENABLE_COMPLEX=OFF`. See #362 about turning off float and complex gy default.)\r\n\r\n4. **Change the main post-push CI server that posts to CDash to use the exact same build as the default builds for the checkin-test-sems.sh script:** This is needed to catch the violations of the [additive test assumption of branches](https://docs.google.com/document/d/1uVQYI2cmNx09fDkHDA136yqDTqayhxqfvjFiuUue7wo/edit#bookmark=id.d1jneh8ubsyn).  This can also be used to alert Trilinos developers when there are failures in the standard CI build or to verify that failures they are seeing are not their doing.  If other post-push CI builds are desired, like non-MPI serial and full release builds, then those can be added as extra CI builds (we just need extra machines for that).\r\n\r\nAfter this Story is complete, then we can create new Stories to get Trilinos developers to use the checkin-test-sems.sh script and to commit to keeping the CI build(s) 100% all the time with \"Stop the Line\" urgency to fix.\r\n\r\n**Definition of Done:**\r\n\r\n* An initial implementation for `load_ci_sems_dev_env.sh` and `checkin-test-sems.sh` that provides a viable CI build based on the SEMS Dev Env.\r\n* Documentation for `load_ci_sems_dev_env.sh` and `checkin-test-sems.sh` has been written and has been reviewed by a few Trilinos developers.\r\n* The post-push CI build of Trilinos uses the same `load_ci_sems_dev_env.sh` env and the same default build(s) as defined in the `checkin-test-sems.sh` script.\r\n* Review the setup and documentation for the `checkin-test.py` script itself to determine what improvements that might help with usability and adoption.\r\n\r\n**Decisions that need to be made:**\r\n\r\n* What default timeout should be selected for pre-push tests (e.g. 3 minutes)?\r\n* Should there just be an MPI_RELEASE_DEBUG_SHARED default build or also some serial build listed in --default-builds?\r\n* What version of GCC and OpenMPI should be used?\r\n* What other set of TPLs really should be added beyond what is provided in the SEMS Dev Env (e.g. a 64-bit build of Scotch without pthreads enabled, see [this comment in #476](https://github.com/trilinos/Trilinos/issues/476#issuecomment-230135162)).\r\n* ???\r\n\r\n**Tasks:**\r\n\r\n1. Create drafts for `load_ci_sems_dev_env.sh` and `checkin-test-sems.sh` [Done]\r\n2. Discuss this Story at a Trilinos Leaders Meeting ...\r\n3. Work #410 to select the updated set of PT packages and TPLs ...\r\n4. Work #362 to disable float and complex by default ...\r\n5. Select the new set (or just one) `--default-builds` for the `checkin-test.py` and therefore the `checkin-test-sems.sh` script ...\r\n6. Update the SEMS Jenkins post-push CI build to source `load_ci_sems_dev_env.sh` and use the same build(s)  as `--default-builds=<build-name>` for `checkin-test-sems.sh` (break off SEMSDevEnvDisables.cmake from SEMSDevEnv.cmake and use to apply the same disables for the outer TribitisCTestDriverCore.cmake driver script)...\r\n7. Create wiki documentation for usage `checkin-test-sems.sh` ...\r\n8. Have updated CI process and documentation reviewed ...\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/256641332","html_url":"https://github.com/trilinos/Trilinos/issues/482#issuecomment-256641332","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/482","id":256641332,"user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars.githubusercontent.com/u/1959736?v=3","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2016-10-27T13:32:00Z","updated_at":"2016-10-27T13:32:00Z","body":"We have a problem with this strategy.  The only GCC compiler that is available on OSX for in the SEMS env is GCC 5.3.0 and the Boost 1.55.0 is not present as shown by:\r\n\r\n```\r\n$ module avail\r\n\r\n...\r\n\r\n--------- /projects/sems/modulefiles/Darwin10.11-x86_64/sems/compiler ----------\r\nsems-gcc/5.3.0      sems-openmpi/1.8.7  sems-python/3.5.2\r\nsems-openmpi/1.10.1 sems-python/2.7.9\r\nsems-openmpi/1.6.5  sems-python/3.4.2\r\n\r\n------------ /projects/sems/modulefiles/Darwin10.11-x86_64/sems/tpl ------------\r\nsems-astroid/1.4.3/base            sems-parmetis/4.0.3/32bit_parallel\r\nsems-beautifulsoup4/4.4.1/base     sems-parmetis/4.0.3/64bit_parallel\r\nsems-boost/1.58.0/base             sems-parmetis/4.0.3/parallel\r\nsems-boost/1.59.0/base             sems-pylint/1.5.4/base\r\nsems-dateutil/2.5.3/base           sems-pyparsing/2.0.3/base\r\nsems-gprof2dot/2015.12.1/base      sems-pytz/2014.10/base\r\nsems-hdf5/1.8.12/base              sems-qd/2.3.15/base\r\nsems-hdf5/1.8.12/parallel          sems-scipy/0.15.1/base\r\nsems-logilab/1.0.2/base            sems-scons/2.3.6/base\r\nsems-matplotlib/1.4.2/base         sems-setuptools/22.0.5/base\r\nsems-netcdf/4.3.2/base             sems-six/1.9.0/base\r\nsems-netcdf/4.3.2/parallel         sems-superlu/4.3/base\r\nsems-numpy/1.9.1/base              sems-zlib/1.2.8/base\r\n\r\n...\r\n```\r\n\r\nTherefore, in order to have one consistent CI env across all platforms is to use GCC 5.3.0 (and Boost 1.58.0 or 1.59.0 but not 1.55.0 which is the current selection).  That may not be so bad.  If we get all warnings out of GCC 5.3.0 they will likely be gone from GCC 4.7.2 as well.  However, there is the problem of people using features of C++11 present in GCC 5.3.0 that is not present with GCC 4.7.2.  Therefore, we would need to run a second post-push CI build that tests with GCC 4.7.2 to make sure things are okay and deal with them quickly.\r\n\r\nWhat do people thing about this?  Is GCC 5.3.0 a valid choice for the CI build for Trilinos?  That is our only option currently.  Otherwise, we are going to need to force OSX developers to test and push from a Linux machine that mounts the SEMS env.\r\n"}}
{"actor":{"display_login":"bartlettroscoe"}}