{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/3499","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3499/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3499/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3499/events","html_url":"https://github.com/trilinos/Trilinos/issues/3499","id":363696804,"node_id":"MDU6SXNzdWUzNjM2OTY4MDQ=","number":3499,"title":"Anasazi tests failing in intel-18.0.2 builds on 'mutrino' and 'cee-rhel6' envs","user":{"login":"fryeguy52","id":14793461,"node_id":"MDQ6VXNlcjE0NzkzNDYx","avatar_url":"https://avatars2.githubusercontent.com/u/14793461?v=4","gravatar_id":"","url":"https://api.github.com/users/fryeguy52","html_url":"https://github.com/fryeguy52","followers_url":"https://api.github.com/users/fryeguy52/followers","following_url":"https://api.github.com/users/fryeguy52/following{/other_user}","gists_url":"https://api.github.com/users/fryeguy52/gists{/gist_id}","starred_url":"https://api.github.com/users/fryeguy52/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fryeguy52/subscriptions","organizations_url":"https://api.github.com/users/fryeguy52/orgs","repos_url":"https://api.github.com/users/fryeguy52/repos","events_url":"https://api.github.com/users/fryeguy52/events{/privacy}","received_events_url":"https://api.github.com/users/fryeguy52/received_events","type":"User","site_admin":false},"labels":[{"id":816763194,"node_id":"MDU6TGFiZWw4MTY3NjMxOTQ=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM","name":"ATDM","color":"ea88a5","default":false},{"id":1145008889,"node_id":"MDU6TGFiZWwxMTQ1MDA4ODg5","url":"https://api.github.com/repos/trilinos/Trilinos/labels/ATDM%20Sev:%20Nonblocker","name":"ATDM Sev: Nonblocker","color":"f9d520","default":false},{"id":1145011874,"node_id":"MDU6TGFiZWwxMTQ1MDExODc0","url":"https://api.github.com/repos/trilinos/Trilinos/labels/PA:%20Linear%20Solvers","name":"PA: Linear Solvers","color":"57d80d","default":false},{"id":287520103,"node_id":"MDU6TGFiZWwyODc1MjAxMDM=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/bug","name":"bug","color":"fc2929","default":true},{"id":333828046,"node_id":"MDU6TGFiZWwzMzM4MjgwNDY=","url":"https://api.github.com/repos/trilinos/Trilinos/labels/pkg:%20Anasazi","name":"pkg: Anasazi","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/trilinos/Trilinos/milestones/47","html_url":"https://github.com/trilinos/Trilinos/milestone/47","labels_url":"https://api.github.com/repos/trilinos/Trilinos/milestones/47/labels","id":3131248,"node_id":"MDk6TWlsZXN0b25lMzEzMTI0OA==","number":47,"title":"Keep promoted \"ATDM\" builds of Trilinos clean","description":"This milestone/epic is to aggregate Issues for new failures in builds with the name \"Trilinos-atdm-XXX\" being posted to the \"ATDM\" CDash Track/Group that is sending out CDash error emails after these builds have been promoted.  These failures have to be fixed in short order.\r\n\r\nThere is no end-date for this milestone/epic.  As new failures are reported, new issues will be added.\r\n","creator":{"login":"bartlettroscoe","id":1959736,"node_id":"MDQ6VXNlcjE5NTk3MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"open_issues":13,"closed_issues":91,"state":"open","created_at":"2018-02-20T22:30:27Z","updated_at":"2019-03-05T19:12:48Z","due_on":null,"closed_at":null},"comments":47,"created_at":"2018-09-25T17:49:22Z","updated_at":"2019-03-11T19:14:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"CC: @trilinos/anasazi , @srajama1  (Trilinos Linear Solvers Product Lead), @bartlettroscoe \r\n \r\n## Next Action Status\r\n\r\n## Description\r\nAs shown in [this query](https://testing.sandia.gov/cdash-dev-view/queryTests.php?project=Trilinos&date=2018-09-24&filtercount=5&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=ATDM&field2=site&compare2=61&value2=mutrino&field3=status&compare3=62&value3=passed&field4=buildstarttime&compare4=83&value4=2018-09-01&field5=testname&compare5=63&value5=Anasazi) the tests:\r\n* `Anasazi_MultiVecTraitsTest2_MPI_4` \r\n* `Anasazi_Epetra_BKS_norestart_test_MPI_4` \r\n\r\nare failing in the builds:\r\n* Trilinos-atdm-mutrino-intel-opt-openmp-HSW\r\n* Trilinos-atdm-mutrino-intel-opt-openmp-KNL\r\n\r\nboth of these tests started failing on 9-22-2018.\r\n\r\nThe test `Anasazi_Epetra_BKS_norestart_test_MPI_4` is also failing in the build `Trilinos-atdm-cee-rhel6-intel-18.0.2-mpich2-3.2-serial-static-opt` for the 'cee-rhel6' inv since it was first set up.\r\n\r\nThe first failures of the test `Anasazi_MultiVecTraitsTest2_MPI_4` on 9/22/2018 is shown [here](https://testing.sandia.gov/cdash-dev-view/testDetails.php?test=55028214&build=3963372) which shows:\r\n\r\n```\r\n Check B_view = CloneViewNonConst(B, ind):\r\n  ind: [0, 2, 4, 6, 8]\r\n  static_cast<size_t> (B_view->getNumVectors ()) = 5 == static_cast<size_t> (ind.size ()) = 5 : passed\r\n  norms of CloneViewNonConst(B, ind): [2.42234, 2.43667, 2.43783, 2.39508, 2.97253]\r\n  B_view_norms[j] = 2.42233923795730766e+00 == normsB1[ind[j]] = 2.42233923795730766e+00 : passed\r\n  B_view_norms[j] = 2.43666989157577962e+00 == normsB1[ind[j]] = 2.43666989157578007e+00 : FAILED ==> /lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/Trilinos/packages/anasazi/tpetra/test/MVOPTester/MultiVecTraitsTest2.cpp:573\r\n...\r\n [FAILED]  (0.158 sec) MultiVecTraits_TpetraSetBlock4_UnitTest\r\n Location: /lscratch1/jenkins/mutrino-slave/workspace/Trilinos-atdm-mutrino-intel-opt-openmp-HSW/SRC_AND_BUILD/Trilinos/packages/anasazi/tpetra/test/MVOPTester/MultiVecTraitsTest2.cpp:434\r\n \r\n\r\nThe following tests FAILED:\r\n    3. MultiVecTraits_TpetraSetBlock4_UnitTest ... \r\n\r\nTotal Time: 1.55 sec\r\n\r\nSummary: total = 4, run = 4, passed = 3, failed = 1\r\n```\r\n\r\nThe first failures of the test `Anasazi_Epetra_BKS_norestart_test_MPI_4` on 9/22/2018 is shown [here](https://testing.sandia.gov/cdash-dev-view/testDetails.php?test=55028175&build=3963372) which shows:\r\n\r\n```\r\nAnasazi_Epetra_BKS_norestart_test.exe: malloc.c:2392: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.\r\nAnasazi_Epetra_BKS_norestart_test.exe: malloc.c:2392: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t)) < __alignof__ (long double) ? __alignof__ (long double) : 2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.\r\nsrun: error: nid00012: tasks 0,3: Segmentation fault\r\nsrun: Terminating job step 11643442.1635\r\nsrun: error: nid00012: task 1: Aborted\r\nslurmstepd: error: *** STEP 11643442.1635 ON nid00012 CANCELLED AT 2018-09-22T07:50:20 ***\r\nsrun: error: nid00012: task 2: Aborted (core dumped)\r\n```\r\n\r\n\r\n\r\nNew commits for this build can be seen [here](https://testing.sandia.gov/cdash-dev-view/viewNotes.php?buildid=3963370#!#note6)\r\n\r\n## Steps to Reproduce\r\n\r\nOne should be able to reproduce this failure on the machine mutrino as described in:\r\n* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md\r\nMore specifically, the commands given for the system mutrino are provided at:\r\n* https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino\r\nThe exact commands to reproduce this issue should be:\r\n```\r\n$ cd <some_build_dir>/\r\n\r\n$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp-HSW\r\n\r\n$ cmake \\\r\n  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \\\r\n  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Anasazi=ON \\\r\n  $TRILINOS_DIR\r\n\r\n$ make -j16\r\n\r\n$ salloc -N 1 -p standard -J $JOB_NAME ctest -j16\r\n```"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/471681638","html_url":"https://github.com/trilinos/Trilinos/issues/3499#issuecomment-471681638","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/3499","id":471681638,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3MTY4MTYzOA==","user":{"login":"bartlettroscoe","id":1959736,"node_id":"MDQ6VXNlcjE5NTk3MzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1959736?v=4","gravatar_id":"","url":"https://api.github.com/users/bartlettroscoe","html_url":"https://github.com/bartlettroscoe","followers_url":"https://api.github.com/users/bartlettroscoe/followers","following_url":"https://api.github.com/users/bartlettroscoe/following{/other_user}","gists_url":"https://api.github.com/users/bartlettroscoe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartlettroscoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartlettroscoe/subscriptions","organizations_url":"https://api.github.com/users/bartlettroscoe/orgs","repos_url":"https://api.github.com/users/bartlettroscoe/repos","events_url":"https://api.github.com/users/bartlettroscoe/events{/privacy}","received_events_url":"https://api.github.com/users/bartlettroscoe/received_events","type":"User","site_admin":false},"created_at":"2019-03-11T19:14:37Z","updated_at":"2019-03-11T19:14:37Z","author_association":"MEMBER","body":"@nmhamster asked:\r\n\r\n> Can you check the KNL jobs are definitely submitting to the correct queue?\r\n\r\nSorry, my bad.  In my manual testing I forgot to add the `-p knl` to the `salloc` command for the KNL build.  But the ctest -S driver does set that correctly.  I am running just the Anaszi build again with:\r\n\r\n```\r\n$ env Trilinos_PACKAGES=Anasazi ./ctest-s-local-test-driver.sh \\\r\n   intel-opt-openmp-HSW intel-opt-openmp-KNL\r\n```\r\n\r\nto see if I can reproduce the behavior I saw over the weekend.  (Don't know why changing from Intel 18.0.2 to Intel 18.0.5 would change how the outer ctest -S driver works.  I will look into this more carefully.)\r\n\r\n\r\n"}}
{"actor":{"display_login":"bartlettroscoe"}}