{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/2473","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2473/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2473/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2473/events","html_url":"https://github.com/trilinos/Trilinos/issues/2473","id":309564149,"number":2473,"title":"Tests Anasazi_Epetra_ModalSolversTester_MPI_4 and Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 failing in Trilinos-atdm-white-ride-cuda-debug and other builds on 3/18/2018","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"id":816763194,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM","name":"ATDM","color":"ea88a5","default":false},{"id":333828046,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Anasazi","name":"Anasazi","color":"f7c6c7","default":false},{"id":287520103,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"mhoemmen","id":926747,"avatar_url":"https://avatars0.githubusercontent.com/u/926747?v=4","gravatar_id":"","url":"https://api.github.com/users/mhoemmen","html_url":"https://github.com/mhoemmen","followers_url":"https://api.github.com/users/mhoemmen/followers","following_url":"https://api.github.com/users/mhoemmen/following{/other_user}","gists_url":"https://api.github.com/users/mhoemmen/gists{/gist_id}","starred_url":"https://api.github.com/users/mhoemmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhoemmen/subscriptions","organizations_url":"https://api.github.com/users/mhoemmen/orgs","repos_url":"https://api.github.com/users/mhoemmen/repos","events_url":"https://api.github.com/users/mhoemmen/events{/privacy}","received_events_url":"https://api.github.com/users/mhoemmen/received_events","type":"User","site_admin":false},"assignees":[{"login":"mhoemmen","id":926747,"avatar_url":"https://avatars0.githubusercontent.com/u/926747?v=4","gravatar_id":"","url":"https://api.github.com/users/mhoemmen","html_url":"https://github.com/mhoemmen","followers_url":"https://api.github.com/users/mhoemmen/followers","following_url":"https://api.github.com/users/mhoemmen/following{/other_user}","gists_url":"https://api.github.com/users/mhoemmen/gists{/gist_id}","starred_url":"https://api.github.com/users/mhoemmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhoemmen/subscriptions","organizations_url":"https://api.github.com/users/mhoemmen/orgs","repos_url":"https://api.github.com/users/mhoemmen/repos","events_url":"https://api.github.com/users/mhoemmen/events{/privacy}","received_events_url":"https://api.github.com/users/mhoemmen/received_events","type":"User","site_admin":false},{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/trilinos/Trilinos/milestones/46","html_url":"https://github.com/trilinos/Trilinos/milestone/46","labels_url":"https://api.github.com/repos/trilinos/Trilinos/milestones/46/labels","id":3108501,"number":46,"title":"Initial cleanup of new ATDM builds of Trilinos","description":"This milestone is to aggregate issues for the initial cleanup of the new ATDM builds of Trilinos.  This initial cleanup is needed to promote these builds from the \"Specialized\" group/track to the \"ATDM\" (and eventually \"Nightly\") group/track so that they trigger regression emails.","creator":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":16,"state":"open","created_at":"2018-02-12T16:35:51Z","updated_at":"2018-03-31T00:27:05Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-03-28T23:08:31Z","updated_at":"2018-03-31T13:35:13Z","closed_at":null,"author_association":"MEMBER","body":"**CC:** @trilinos/anasazi, @mhoemmen \r\n\r\n## Description\r\n\r\nThe tests:\r\n\r\n* `Anasazi_Epetra_ModalSolversTester_MPI_4`\r\n* `Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4`\r\n\r\nfailed in `Trilinos-atdm-hansen-shiller-cuda-debug` build on 'ride' as shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3398198\r\n* https://testing-vm.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=3398699\r\n\r\nThis build is targeted to be an auto PR build for Trilinos (see #2464) so we desire to clean up this build more quickly.\r\n\r\nIntrestingly, these tests did **not** fail in what should be the idential `Trilinos-atdm-hansen-shiller-cuda-debug` build on the identical machine 'white' as shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/index.php?project=Trilinos&parentid=3398098\r\n* https://testing-vm.sandia.gov/cdash/viewTest.php?buildid=3398630\r\n\r\nStrangely, those tests **did** fail on `Trilinos-atdm-hansen-shiller-cuda-debug` build on 'white' yestrday shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/viewTest.php?buildid=3396229\r\n\r\n**A) Anasazi_Epetra_ModalSolversTester_MPI_4:**\r\n\r\nTest failing test `Anasazi_Epetra_ModalSolversTester_MPI_4` today with details shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/testDetails.php?test=44732780&build=3398699\r\n\r\nshowed the failure:\r\n\r\n```\r\n************* Householder Apply Test *************\r\n\r\n             orthonorm error of V: 7.08978e-16\r\n            orthonorm error of VQ: 0.375867\r\nERROR:  V*Q failed.\r\n    orthonorm error of applyHouse: 0.375867\r\nERROR:  applyHouse failed.\r\n        error(VQ - house(V,H,tau): 2.64481e-16\r\n\r\n************* DirectSolver Test *************\r\n```\r\n\r\nLooking at all of the builds today that ran that test shown at:\r\n\r\n* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-28&limit=0&filtercount=1&showfilters=1&field1=testname&compare1=61&value1=Anasazi_Epetra_ModalSolversTester_MPI_4\r\n\r\nthis test fails in the same way (i.e. a numerical problem) on the builds `Linux-gcc-4.8.4-MPI_RELEASE_12.12.1` and `Linux-gcc-4.8.4-MPI_RELEASE_12.12.1_SHARED` on the machine `hansel.sandia.gov` so this problem is not isolated to ATDM builds of Trilinos.\r\n\r\nAlso note that this test failed for the ATDM builds `Trilinos-atdm-white-ride-gnu-opt-openmp` and `Trilinos-atdm-white-ride-gnu-opt-openmp` with segfaults, but that is already being addressed by #2454 and is likely unrelated.\r\n\r\n**B) Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4:**\r\n\r\nThe failing test `Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4` today with details shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/testDetails.php?test=44732781&build=3398699\r\n\r\nshowed:\r\n\r\n```\r\nAnasazi in Trilinos 12.13 (Dev)\r\n\r\n Generating Y1,Y2 for project() : testing... \r\n   || <Y1,Y1> - I || : 6.47718e-16\r\n   || <Y2,Y2> - I || : 7.20309e-16\r\n   || <X1,Y2> ||     : 1.64775e-16\r\n   || <X1b,Y2> ||     : 6.9984e-15\r\n\r\np=3: *** Caught standard std::exception of type 'std::runtime_error' :\r\n\r\n /home/jenkins/ride/workspace/Trilinos-atdm-white-ride-cuda-debug/SRC_AND_BUILD/Trilinos/packages/anasazi/epetra/test/OrthoManager/cxx_gentest.cpp:274:\r\n \r\n Throw number = 1\r\n \r\n Throw test that evaluated to true: err > TOL\r\n \r\n New X1 did not meet tolerance: orthog(X1,Y2) == 0.547032\r\n```\r\n\r\nLooking at all of the builds today that ran that test shown at:\r\n\r\n* https://testing.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-28&limit=0&filtercount=1&showfilters=1&field1=testname&compare1=61&value1=Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4\r\n\r\nyou can see that this test also failed in a similar (numerical) way in the builds `Linux-gcc-4.9.3-Sierra_MPI_release_DEV_ETI_SERIAL-ON_OPENMP-ON_PTHREAD-OFF_CUDA-OFF_COMPLEX-ON` and `Linux-GCC-4.9.3-openmpi-1.8.7_Debug_DEV_Werror` so it looks like this problem is not isolated to ATDM builds of Trilinos.  Note that one of those is a \"Sierra' build of Trilinos.\r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/377693434","html_url":"https://github.com/trilinos/Trilinos/issues/2473#issuecomment-377693434","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2473","id":377693434,"user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2018-03-31T13:35:13Z","updated_at":"2018-03-31T13:35:13Z","author_association":"MEMBER","body":"These randomly failing tests triggered the following CDash error email for the newly promoted build ??? this morning.\r\n\r\nCan I go ahead and disable these randomly failing test in these builds?  The tests will only be disabled for these builds and not others where the test is passing consistently.\r\n\r\n----\r\n\r\nFrom: CDash [mailto:trilinos-regression@sandia.gov]\r\nSent: Saturday, March 31, 2018 2:48 AM\r\nTo: Bartlett, Roscoe A <rabartl@sandia.gov>\r\nSubject: FAILED (t=2): Trilinos/Anasazi - Trilinos-atdm-white-ride-gnu-debug-openmp - ATDM\r\n\r\nA submission to CDash for the project Trilinos has failing tests.\r\nYou have been identified as one of the authors who have checked in changes that are part of this submission or you are listed in the default contact list.\r\n\r\nDetails on the submission can be found at https://testing.sandia.gov/cdash/buildSummary.php?buildid=3474500\r\n\r\nProject: Trilinos\r\nSubProject: Anasazi\r\nSite: white\r\nBuild Name: Trilinos-atdm-white-ride-gnu-debug-openmp\r\nBuild Time: 2018-03-31T06:45:53 UTC\r\nType: ATDM\r\nTests failing: 2\r\n\r\n*Tests failing*\r\nAnasazi_Epetra_ModalSolversTester_MPI_4 (https://testing.sandia.gov/cdash/testDetails.php?test=46065301&build=3474500)\r\nAnasazi_Epetra_OrthoManagerGenTester_0_MPI_4 (https://testing.sandia.gov/cdash/testDetails.php?test=46065302&build=3474500)\r\n\r\n\r\n-CDash on testing.sandia.gov\r\n"}}
{"actor":{"display_login":"bartlettroscoe"}}