{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/2620","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2620/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2620/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2620/events","html_url":"https://github.com/trilinos/Trilinos/issues/2620","id":316901222,"number":2620,"title":"Clean up builds and tests for all Primary Tested packages for CUDA build on white/ride","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"id":310381463,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Framework","name":"Framework","color":"bfd4f2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-23T17:07:06Z","updated_at":"2018-04-25T18:55:59Z","closed_at":null,"author_association":"MEMBER","body":"**CC:** @trilinos/framework\r\n\r\n## Description\r\n\r\nThis issue is a placeholder for cleaning up the builds and tests for all of the Primary Tested Trilinos packages for the `cuda-debug` build on `white/ride` as described in https://github.com/trilinos/Trilinos/issues/2464#issuecomment-383646788.\r\n\r\nAs of 4/23/2018, the build `Trilinos-atdm-white-ride-cuda-debug-pt-all-at-once` on `white` for all 53 of the current Primary Tested Trilinos packages passed the configure but has several build and runtime test failures as shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3451394\r\n\r\nThis shows build failures for the packages:\r\n\r\n* ROL (@trilinos/rol)\r\n* MiniTensor (@lxmota)\r\n* Zoltan (@trilinos/zoltan)\r\n* TrilinosCouplings (???)\r\n* ShyLU_DD (@trilinos/shylu )\r\n* Domi (@trilinos/domi)\r\n\r\nAnd it shows test failures for the packages:\r\n\r\n* Stokhos (@trilinos/stokhos)\r\n* ROL\r\n* Zoltan\r\n* TrilinosCouplings\r\n* Piro (@trilinos/piro)\r\n* FEI (@trilinos/fei)\r\n\r\n## Steps to Reproduce\r\n\r\nAnyone should be able to reproduce any of these package failures on 'white`'(SON) or 'ride' (SRN) as:\r\n\r\n```\r\n$ cd <some_build_dir>/\r\n\r\n$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh cuda-debug\r\n\r\n$ cmake \\\r\n  -GNinja \\\r\n  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnvSettungs.cmake \\\r\n  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<PackageName>=ON \\\r\n  $TRILINOS_DIR\r\n\r\n$ ninja -j16 -k 999999\r\n\r\n$ bsub -x -Is -q rhel7F -n 16 ctest -j16\r\n```\r\n\r\nFor more details, see:\r\n\r\n* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/384396729","html_url":"https://github.com/trilinos/Trilinos/issues/2620#issuecomment-384396729","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2620","id":384396729,"user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2018-04-25T18:55:59Z","updated_at":"2018-04-25T18:55:59Z","author_association":"MEMBER","body":"> I've been looking into the stokhos failures. They appear to be seg faults associated with calls to the LAPACK function STEQR through the Teuchos::LAPACK interface. Were the issues with BLAS/LAPACK ever resolved on this machine?\r\n\r\n@etphipp,\r\n\r\nAs documented in #1208 last year, Tech-X contractors tracked this down to what looks like a compiler defect in passing some arguments between C++ and C or Fortran.  But most of those failures appear to be occurring with compiler optimization turned on as evidenced by the numerous values we see in the 'opt' builds on this system as documented in #2454.  But we did see an LAPACK test failing even in the 'debug' build as described in #2410.  Given all of those failing Belos and Anasazi tests (which use BLAS and LAPACK) for the 'opt' builds, I don't understand how anyone is using the solvers in Trilinos on this system (but somehow they are).\r\n\r\nThe solution we were told was to simply wait for the next version of ATS-2 which should be what exists on the new testbed machine waterman.  That was the purpose of the placeholder issue #1675.  Issues with that system can be discussed in the protected issue:\r\n\r\n* https://sems-atlassian-srn.sandia.gov/browse/CDOFA-24\r\n\r\nGiven all of that, I don't know that we should try too hard to address failures related to mixed-language calls on this current Power8 system on 'ride' and 'white'\r\n\r\nCC: @nmhamster "}}
{"actor":{"display_login":"bartlettroscoe"}}