{"action":"created","issue":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/227","repository_url":"https://api.github.com/repos/trilinos/Trilinos","labels_url":"https://api.github.com/repos/trilinos/Trilinos/issues/227/labels{/name}","comments_url":"https://api.github.com/repos/trilinos/Trilinos/issues/227/comments","events_url":"https://api.github.com/repos/trilinos/Trilinos/issues/227/events","html_url":"https://github.com/trilinos/Trilinos/issues/227","id":141884725,"number":227,"title":"Tpetra range issues in Kokkos subview with new view","user":{"login":"bathmatt","id":14926624,"avatar_url":"https://avatars.githubusercontent.com/u/14926624?v=3","gravatar_id":"","url":"https://api.github.com/users/bathmatt","html_url":"https://github.com/bathmatt","followers_url":"https://api.github.com/users/bathmatt/followers","following_url":"https://api.github.com/users/bathmatt/following{/other_user}","gists_url":"https://api.github.com/users/bathmatt/gists{/gist_id}","starred_url":"https://api.github.com/users/bathmatt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bathmatt/subscriptions","organizations_url":"https://api.github.com/users/bathmatt/orgs","repos_url":"https://api.github.com/users/bathmatt/repos","events_url":"https://api.github.com/users/bathmatt/events{/privacy}","received_events_url":"https://api.github.com/users/bathmatt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/trilinos/Trilinos/labels/Tpetra","name":"Tpetra","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"mhoemmen","id":926747,"avatar_url":"https://avatars.githubusercontent.com/u/926747?v=3","gravatar_id":"","url":"https://api.github.com/users/mhoemmen","html_url":"https://github.com/mhoemmen","followers_url":"https://api.github.com/users/mhoemmen/followers","following_url":"https://api.github.com/users/mhoemmen/following{/other_user}","gists_url":"https://api.github.com/users/mhoemmen/gists{/gist_id}","starred_url":"https://api.github.com/users/mhoemmen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhoemmen/subscriptions","organizations_url":"https://api.github.com/users/mhoemmen/orgs","repos_url":"https://api.github.com/users/mhoemmen/repos","events_url":"https://api.github.com/users/mhoemmen/events{/privacy}","received_events_url":"https://api.github.com/users/mhoemmen/received_events","type":"User","site_admin":false},"milestone":null,"comments":102,"created_at":"2016-03-18T14:37:32Z","updated_at":"2016-05-03T02:33:55Z","closed_at":null,"body":"@mhoemmen @crtrott @hcedwar \r\n\r\nI'm getting this error when I have bounds checking on\r\n\r\n```\r\nKokkos::subview bounds errorp=1 | *********** Caught Exception std::exception: Begin Error Report ***********\r\np=1 | /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2136:\r\np=1 |\r\np=1 | Throw number = 1\r\np=1 |\r\np=1 | Throw test that evaluated to true: static_cast<size_t> (imports.dimension_0 ()) < totalNumImportPackets\r\np=1 |\r\np=1 | Tpetra::Distributor::doPosts(3 args): The 'imports' array must have enough entries to hold the expected number of import packets.  imports.dimension_0() = 0 < totalNumImportPackets = 63.\r\np=1 | ************ Caught Exception std::exception: End Error Report ************\r\n--------------------------------------------------------------------------\r\nmpirun noticed that process rank 0 with PID 5293 on node hansen02 exited on signal 11 (Segmentation fault).\r\n```\r\n\r\nI've run this code in serial and parallel through valgrind , it is clean\r\nI've run it through cuda-memcheck in parallel and it doesn't show anything except for the exception shown below.\r\nI've run it through cuda-gdb in parallel, I've attached the tracebacks.\r\n\r\nI think this is an issue with tpetra now. This is on Hansen, I can make the build public. \r\n\r\n On rank 1\r\n```\r\n(cuda-gdb) where\r\n#0  __cxxabiv1::__cxa_throw (obj=0x1f81f6c0,\r\n    tinfo=0x1d4b1b60 <_ZTISt13runtime_error@@GLIBCXX_3.4>,\r\n    dest=0x49b40c0 <_ZNSt13runtime_errorD1Ev@plt>)\r\n    at ../../.././libstdc++-v3/libsupc++/eh_throw.cc:62\r\n#1  0x000000000843950e in Tpetra::Distributor::doPosts<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (this=\r\n    0x1f7a9610, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2131\r\n#2  0x000000000842fb78 in Tpetra::Distributor::doPostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (\r\n    this=0x1f7a9610, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2009\r\n#3  0x000000000842b41a in Tpetra::Distributor::doReversePostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::Cuda> > (this=0x1f7a9610, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distribut---Type <return> to continue, or q <return> to quit---\r\nor.hpp:2803\r\n#4  0x0000000008429bc2 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransferNew\r\n    (this=0x1f7aea40, src=..., CM=Tpetra::REPLACE, numSameIDs=231,\r\n    permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,\r\n    exportLIDs_=..., distor=...,\r\n    revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:856\r\n#5  0x0000000008428519 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransfer (\r\n    this=0x1f7aea40, src=..., CM=Tpetra::REPLACE, numSameIDs=231,\r\n    permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,\r\n    exportLIDs_=..., distor=...,\r\n    revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:397\r\n#6  0x00000000084270bd in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doImport (\r\n    this=0x1f7aea40, source=..., exporter=..., CM=Tpetra::REPLACE)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObjec---Type <return> to continue, or q <return> to quit---\r\nt_def.hpp:333\r\n#7  0x0000000006b7b330 in panzer::DOFManager<int, int>::buildGlobalUnknowns (\r\n    this=0x1f7a3780, geomPattern=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:505\r\n#8  0x0000000006b7a074 in panzer::DOFManager<int, int>::buildGlobalUnknowns (\r\n    this=0x1f7a3780)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:317\r\n#9  0x00000000064ad919 in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer<panzer::DOFManager<int, int> > (this=0x7fffffff6a80, mpiComm=...,\r\n    physicsBlocks=..., connMngr=..., fieldOrder=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:148\r\n#10 0x00000000064ac4ba in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer (this=0x7fffffff6a80, mpiComm=..., physicsBlocks=..., connMngr=...,\r\n    fieldOrder=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:66\r\n#11 0x0000000004bef189 in drekar::UniqueGlobalIndexerFactory<int>::buildUniqueGlobalIndexer (this=0x7fffffff6cb0, mpiComm=..., physicsBlocks_exp=...,\r\n```\r\non rank 0\r\n```\r\n#1  0x00007fffeca13e15 in abort () from /lib64/libc.so.6\r\n#2  0x0000000008d2e9c2 in Kokkos::Impl::host_abort (\r\n    message=0x9ec8f16 <Kokkos::Experimental::(anonymous namespace)::AllowPadding+25640> \"Kokkos::subview bounds error\")\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.cpp:64\r\n#3  0x0000000004ff6eca in Kokkos::abort (\r\n    message=0x9ec8f16 <Kokkos::Experimental::(anonymous namespace)::AllowPadding+25640> \"Kokkos::subview bounds error\")\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/Kokkos_Error.hpp:74\r\n#4  0x0000000007ca65a2 in error<0ul, std::pair<unsigned long, unsigned long> >\r\n    (dim=..., this=0x7fffffff4630)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:552\r\n#5  Kokkos::Experimental::Impl::SubviewExtents<1u, 1u>::SubviewExtents<0ul, std::pair<unsigned long, unsigned long> > (this=0x7fffffff4630, dim=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:587\r\n#6  0x00000000084434ba in Kokkos::Experimental::Impl::ViewMapping<void, Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, std::pair<unsigned lo---Type <return> to continue, or q <return> to quit---\r\nng, unsigned long> >::assign<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> > >(Kokkos::Experimental::Impl::ViewMapping<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> ><void> >&, Kokkos::Experimental::Impl::ViewMapping<Kokkos::Experimental::ViewTraits<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> ><void> > const&, std::pair<unsigned long, unsigned long>) (dst=..., src=...,\r\n    args#0=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/impl/KokkosExp_ViewMapping.hpp:2788\r\n#7  0x00000000084417c5 in Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >::View<int const*<Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, std::pair<unsigned long, unsigned long>> (this=0x7fffffff49e0, src_view=..., arg0=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/KokkosExp_View.hpp:1160\r\n#8  0x000000000843f2bf in Kokkos::Experimental::subview<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u>, std::pair<unsigned long, unsigned long> > (src=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/src/KokkosExp_View.hpp:1462\r\n---Type <return> to continue, or q <return> to quit---\r\n#9  0x000000000843d56e in Kokkos::Compat::subview_offset<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, unsigned long> (view=..., offset=0,\r\n    size=63)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/teuchos/kokkoscompat/src/KokkosCompat_View.hpp:221\r\n#10 0x0000000008439d5d in Tpetra::Distributor::doPosts<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (this=\r\n    0x2b881130, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2263\r\n#11 0x000000000842fb78 in Tpetra::Distributor::doPostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace> > > (\r\n    this=0x2b881130, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2009\r\n#12 0x000000000842b41a in Tpetra::Distributor::doReversePostsAndWaits<Kokkos::Experimental::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Cuda, Kokkos::CudaUVMSpace>, Kokkos::MemoryTraits<0u> >, Kokkos::Experimental::View<int*---Type <return> to continue, or q <return> to quit---\r\n, Kokkos::Cuda> > (this=0x2b881130, exports=..., numPackets=3, imports=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Distributor.hpp:2803\r\n#13 0x0000000008429bc2 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransferNew\r\n    (this=0x2b875ec0, src=..., CM=Tpetra::REPLACE, numSameIDs=10,\r\n    permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,\r\n    exportLIDs_=..., distor=...,\r\n    revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:856\r\n#14 0x0000000008428519 in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doTransfer (\r\n    this=0x2b875ec0, src=..., CM=Tpetra::REPLACE, numSameIDs=10,\r\n    permuteToLIDs_=..., permuteFromLIDs_=..., remoteLIDs_=...,\r\n    exportLIDs_=..., distor=...,\r\n    revOp=Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::DoReverse)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:397\r\n#15 0x00000000084270bd in Tpetra::DistObject<int, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaUVMSpace>, false>::doImport (\r\n---Type <return> to continue, or q <return> to quit---\r\n    this=0x2b875ec0, source=..., exporter=..., CM=Tpetra::REPLACE)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_DistObject_def.hpp:333\r\n#16 0x0000000006b7b330 in panzer::DOFManager<int, int>::buildGlobalUnknowns (\r\n    this=0x2b87f8c0, geomPattern=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:505\r\n#17 0x0000000006b7a074 in panzer::DOFManager<int, int>::buildGlobalUnknowns (\r\n    this=0x2b87f8c0)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/dof-mgr/src/Panzer_DOFManager_impl.hpp:317\r\n#18 0x00000000064ad919 in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer<panzer::DOFManager<int, int> > (this=0x7fffffff6a90, mpiComm=...,\r\n    physicsBlocks=..., connMngr=..., fieldOrder=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:148\r\n#19 0x00000000064ac4ba in panzer::DOFManagerFactory<int, int>::buildUniqueGlobalIndexer (this=0x7fffffff6a90, mpiComm=..., physicsBlocks=..., connMngr=...,\r\n    fieldOrder=...)\r\n    at /home/mbetten/Trilinos/Trilinos/packages/panzer/disc-fe/src/Panzer_DOFManagerFactory_impl.hpp:66\r\n#20 0x0000000004bef189 in drekar::UniqueGlobalIndexerFactory<int>::buildUniqueGlobalIndexer (this=0x7fffffff6cc0, mpiComm=..., physicsBlocks_exp=..., \r\n```"},"comment":{"url":"https://api.github.com/repos/trilinos/Trilinos/issues/comments/216421624","html_url":"https://github.com/trilinos/Trilinos/issues/227#issuecomment-216421624","issue_url":"https://api.github.com/repos/trilinos/Trilinos/issues/227","id":216421624,"user":{"login":"crtrott","id":9490481,"avatar_url":"https://avatars.githubusercontent.com/u/9490481?v=3","gravatar_id":"","url":"https://api.github.com/users/crtrott","html_url":"https://github.com/crtrott","followers_url":"https://api.github.com/users/crtrott/followers","following_url":"https://api.github.com/users/crtrott/following{/other_user}","gists_url":"https://api.github.com/users/crtrott/gists{/gist_id}","starred_url":"https://api.github.com/users/crtrott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crtrott/subscriptions","organizations_url":"https://api.github.com/users/crtrott/orgs","repos_url":"https://api.github.com/users/crtrott/repos","events_url":"https://api.github.com/users/crtrott/events{/privacy}","received_events_url":"https://api.github.com/users/crtrott/received_events","type":"User","site_admin":false},"created_at":"2016-05-03T02:33:55Z","updated_at":"2016-05-03T02:33:55Z","body":"Btw. the name is a bit misleading. We do not require CUDA-RDMA: what we require is that MPI accepts device pointers. This actually works without proper GPU-Direct. So if someone could try that out (just the OSU benchmark for GPUs should suffice, or I can provide a little example if you want). "}}