{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/2311","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2311/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2311/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2311/events","html_url":"https://github.com/trilinos/Trilinos/issues/2311","id":301129645,"number":2311,"title":"MueLu_ParameterListInterpreterXXX tests appear to be randomly failing with GCC 7.2.0 build","user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"labels":[{"id":816763194,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM","name":"ATDM","color":"ea88a5","default":false},{"id":289606071,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/MueLu","name":"MueLu","color":"5319e7","default":false},{"id":287520103,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/bug","name":"bug","color":"fc2929","default":true},{"id":318642986,"url":"https://api.github.com/repos/trilinos/Trilinos/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/trilinos/Trilinos/milestones/47","html_url":"https://github.com/trilinos/Trilinos/milestone/47","labels_url":"https://api.github.com/repos/trilinos/Trilinos/milestones/47/labels","id":3131248,"number":47,"title":"Keep promoted \"ATDM\" builds of Trilinos clean","description":"This milestone/epic is to aggregate Issues for new failures in builds with the name \"Trilinos-atdm-XXX\" being posted to the \"ATDM\" CDash Track/Group that is sending out CDash error emails after these builds have been promoted.  These failures have to be fixed in short order.\r\n\r\nThere is no end-date for this milestone/epic.  As new failures are reported, new issues will be added.\r\n","creator":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":6,"state":"open","created_at":"2018-02-20T22:30:27Z","updated_at":"2018-05-15T00:40:05Z","due_on":null,"closed_at":null},"comments":31,"created_at":"2018-02-28T18:18:03Z","updated_at":"2018-05-15T12:11:15Z","closed_at":null,"author_association":"MEMBER","body":"**CC:** @trilinos/muelu \r\n\r\n## Next Action Status\r\n\r\n`MueLu_ParameterListInterpreterXXX` tests failed (randomly) 5 times in the `Trilinos-atdm-sems-gcc-7-2-0` build between 2018-03-15 and 2018-04-06.  Next: Fix cause of random failures or disable all `MueLu_ParameterListInterpreterXXX` tests in this build ...\r\n\r\n## Description\r\n\r\nThe tests `MueLu_ParameterListInterpreterTpetra_MPI_1` and `MueLu_ParameterListInterpreterTpetraHeavy_MPI_1` appear to be failing randomly from the GCC build `Trilinos-atdm-sems-gcc-7-2-0` as shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&date=2018-03-12&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=4&showfilters=1&filtercombine=and&field1=testname&compare1=65&value1=MueLu_ParameterListInterpreter&field2=buildstarttime&compare2=84&value2=now&field3=status&compare3=61&value3=failed&field4=buildname&compare4=61&value4=Trilinos-atdm-sems-gcc-7-2-0\r\n\r\n(sort by \"Build Name\", then \"Test Name\", then \"Build Time\").\r\n\r\nFor example, the test `MueLu_ParameterListInterpreterTpetra_MPI_1` is failed as shown at:\r\n\r\n* https://testing.sandia.gov/cdash/testDetails.php?test=44972713&build=3410578\r\n\r\nIt shows the failures:\r\n\r\n```\r\nTesting: MLParameterListInterpreter/MLsmoother1.xml\r\nBinary files Output/MLsmoother1_tpetra.gold_filtered and Output/MLsmoother1_tpetra.out_filtered differ\r\n\r\n...\r\n\r\n\r\nTesting: MLParameterListInterpreter/MLunsmoothed1.xml\r\n--- Output/MLunsmoothed1_tpetra.gold_filtered\t2018-02-28 09:24:11.570595286 -0700\r\n+++ Output/MLunsmoothed1_tpetra.out_filtered\t2018-02-28 09:24:11.582595434 -0700\r\n@@ -280,7 +280,7 @@\r\n  matrixmatrix: kernel params -> \r\n   [empty list]\r\n  \r\n- Setup Smoother (MueLu::Amesos2Smoother{type = <ignored>})\r\n+ Setup Smoother (MueLu::Amesos2Smoother{type = Klu})\r\n  keep smoother data = 0   [default]\r\n  PreSmoother data = Teuchos::RCP<MueLu::SmootherPrototype<ignored> >{ptr=0,node=0,strong_count=0,weak_count=0}   [default]\r\n  PostSmoother data = Teuchos::RCP<MueLu::SmootherPrototype<ignored> >{ptr=0,node=0,strong_count=0,weak_count=0}   [default]\r\n@@ -311,6 +311,6 @@\r\n \r\n Smoother (level 3) both : \"Ifpack2::Relaxation\": {Initialized: true, Computed: true, Type: Symmetric Gauss-Seidel, sweeps: 2, damping factor: 1, Global matrix dimensions: [371, 371], Global nnz: 1111}\r\n \r\n-Smoother (level 4) pre  : <Direct> solver interface\r\n+Smoother (level 4) pre  : KLU2 solver interface\r\n Smoother (level 4) post : no smoother\r\n \r\nMLParameterListInterpreter/MLunsmoothed1.xml : failed\r\nTesting: MLParameterListInterpreter/MLsmoother4.xml\r\n--- Output/MLsmoother4_tpetra.gold_filtered\t2018-02-28 09:24:12.279604031 -0700\r\n+++ Output/MLsmoother4_tpetra.out_filtered\t2018-02-28 09:24:12.277604006 -0700\r\n@@ -284,7 +284,7 @@\r\n  matrixmatrix: kernel params -> \r\n   [empty list]\r\n  \r\n- Setup Smoother (MueLu::Amesos2Smoother{type = <ignored>})\r\n+ Setup Smoother (MueLu::Amesos2Smoother{type = Klu})\r\n  keep smoother data = 0   [default]\r\n  PreSmoother data = Teuchos::RCP<MueLu::SmootherPrototype<ignored> >{ptr=0,node=0,strong_count=0,weak_count=0}   [default]\r\n  PostSmoother data = Teuchos::RCP<MueLu::SmootherPrototype<ignored> >{ptr=0,node=0,strong_count=0,weak_count=0}   [default]\r\nMLParameterListInterpreter/MLsmoother4.xml : failed\r\n```\r\n\r\n## Steps to Reproduce\r\n\r\nUsing the `do-configure` script:\r\n\r\n```\r\n#!/bin/bash\r\ncmake \\\r\n-DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/sems/atdm/SEMSATDMSettings.cmake,cmake/std/MpiReleaseDebugSharedPtSettings.cmake,cmake/std/BasicCiTestingSettings.cmake \\\r\n-DDART_TESTING_TIMEOUT:STRING=300.0 \\\r\n-DTrilinos_ENABLE_TESTS:BOOL=ON \\\r\n-DCTEST_BUILD_FLAGS=-j10 \\\r\n-DCTEST_PARALLEL_LEVEL=10 \\\r\n\"$@\" \\\r\n$TRILINOS_DIR\r\n```\r\n\r\nAnyone should be able to reproduce these builds and run these tests on any SNL COE RHEL6 machine as shown below:\r\n\r\n```\r\n$ cd <some_build_dir>/\r\n\r\n$ source $TRILINOS_DIR/cmake/std/sems/atdm/load_atdm_7.2_dev_env.sh\r\n\r\n$ ./do-configure -DTrilinos_ENABLE_MueLu=ON\r\n\r\n$ make -j16\r\n\r\n$ ctest -j16\r\n```\r\n\r\nHowever, given that tests seem to be randomly failing, it may be hard to reproduce these failures.\r\n\r\n## Related Issues \r\n\r\n* Relates to: #2365 \r\n"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/389144168","html_url":"https://github.com/trilinos/Trilinos/issues/2311#issuecomment-389144168","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/2311","id":389144168,"user":{"login":"bartlettroscoe","id":1959736,"avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T12:11:15Z","updated_at":"2018-05-15T12:11:15Z","author_association":"MEMBER","body":"FYI:\r\n\r\nAs shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercount=4&showfilters=1&field1=buildname&compare1=61&value1=Trilinos-atdm-sems-gcc-7-2-0&field2=testname&compare2=65&value2=MueLu_ParameterListInterpreter&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=84&value4=now\r\n\r\nthe test `MueLu_ParameterListInterpreterTptra_MPI_1` or `MueLu_ParameterListInterpreterTpetraHeavy_MPI_1` failed four times in the last 15 days, just in this one  `Trilinos-atdm-sems-gcc-7-2-0` build.\r\n\r\nAnd as shown at:\r\n\r\n* https://testing-vm.sandia.gov/cdash/queryTests.php?project=Trilinos&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercombine=and&filtercount=6&showfilters=1&filtercombine=and&field1=buildname&compare1=65&value1=Trilinos-atdm-&field2=testname&compare2=65&value2=MueLu_ParameterListInterpreter&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=84&value4=now&field5=buildname&compare5=64&value5=Trilinos-atdm-toss3&field6=status&compare6=62&value6=notrun\r\n\r\none of these tests failed inthe builds `Trilinos-atdm-white-ride-cuda-debug` and `Trilinos-atdm-white-ride-cuda-opt` on `white` and `ride` well several times in the last 15 days.\r\n\r\nThis issue has been open for more than 2.5 months now.  That is enough time to have fixed this issue, if it was going to be fixed.  Therefore, it is time to disable these tests in the `Trilinos-atdm-sems-gcc-7-2-0`, `Trilinos-atdm-white-ride-cuda-debug` and `Trilinos-atdm-white-ride-cuda-opt`.  And if this tests is seen to fail in the same way in other builds, we will disable it in those builds as well.\r\n\r\nAs long as the auto PR builds run these tests and other tests cover the core functionality of MueLu, then disabling these tests should not be so bad.  But if core functionality of MueLu is only covered in these tests, then these should be fixed.\r\n"}}
{"actor":{"display_login":"bartlettroscoe"}}