{"action":"opened","issue":{"url":"https://api.github.com/repos/kokkos/kokkos/issues/1904","repository_url":"https://api.github.com/repos/kokkos/kokkos","labels_url":"https://api.github.com/repos/kokkos/kokkos/issues/1904/labels{/name}","comments_url":"https://api.github.com/repos/kokkos/kokkos/issues/1904/comments","events_url":"https://api.github.com/repos/kokkos/kokkos/issues/1904/events","html_url":"https://github.com/kokkos/kokkos/issues/1904","id":380935397,"node_id":"MDU6SXNzdWUzODA5MzUzOTc=","number":1904,"title":"CudaDeviceSynchronize() error after updating Kokkos","user":{"login":"stanmoore1","id":18146301,"node_id":"MDQ6VXNlcjE4MTQ2MzAx","avatar_url":"https://avatars2.githubusercontent.com/u/18146301?v=4","gravatar_id":"","url":"https://api.github.com/users/stanmoore1","html_url":"https://github.com/stanmoore1","followers_url":"https://api.github.com/users/stanmoore1/followers","following_url":"https://api.github.com/users/stanmoore1/following{/other_user}","gists_url":"https://api.github.com/users/stanmoore1/gists{/gist_id}","starred_url":"https://api.github.com/users/stanmoore1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stanmoore1/subscriptions","organizations_url":"https://api.github.com/users/stanmoore1/orgs","repos_url":"https://api.github.com/users/stanmoore1/repos","events_url":"https://api.github.com/users/stanmoore1/events{/privacy}","received_events_url":"https://api.github.com/users/stanmoore1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-14T23:38:53Z","updated_at":"2018-11-14T23:38:53Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I tried updating Kokkos in SPARTA to latest master, but one of the regression tests is failing with CUDA:\r\n```\r\nwarning: Cuda API error detected: cudaDeviceSynchronize returned (0x4d)\r\n\r\nterminate called after throwing an instance of 'std::runtime_error'\r\n  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:119\r\nTraceback functionality not available\r\n```\r\n\r\nIt passed fine previously. Any ideas? Here is a stack trace running on V100:\r\n\r\n```\r\nThread 1 \"spa_kokkos_cuda\" received signal SIGABRT, Aborted.\r\n0x00007ffff69efaf0 in raise () from /lib64/libc.so.6\r\n(cuda-gdb) where\r\n#0  0x00007ffff69efaf0 in raise () from /lib64/libc.so.6\r\n#1  0x00007ffff69f1e6c in abort () from /lib64/libc.so.6\r\n#2  0x00007ffff6d80774 in __gnu_cxx::__verbose_terminate_handler ()\r\n    at ../../.././libstdc++-v3/libsupc++/vterminate.cc:95\r\n#3  0x00007ffff6d7b504 in __cxxabiv1::__terminate (handler=<optimized out>)\r\n    at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:47\r\n#4  0x00007ffff6d7b5c0 in std::terminate ()\r\n    at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:57\r\n#5  0x00007ffff6d7ba6c in __cxxabiv1::__cxa_throw (obj=<optimized out>,\r\n    tinfo=0x7ffff6ee9a08 <typeinfo for std::runtime_error>,\r\n    dest=0x7ffff6d9c030 <std::runtime_error::~runtime_error()>)\r\n    at ../../.././libstdc++-v3/libsupc++/eh_throw.cc:93\r\n#6  0x0000000010589a20 in Kokkos::Impl::throw_runtime_exception (msg=...)\r\n    at ../../lib/kokkos/core/src/impl/Kokkos_Error.cpp:72\r\n#7  0x00000000105964e8 in Kokkos::Impl::cuda_internal_error_throw (\r\n    e=cudaErrorIllegalAddress,\r\n    name=0x1061cc70 <Kokkos::(anonymous namespace)::AllowPadding+58> \"cudaDeviceSynchronize()\",\r\n    file=0x1061cc38 <Kokkos::(anonymous namespace)::AllowPadding+2> \"../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp\", line=119)\r\n    at ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:129\r\n#8  0x000000001001584c in Kokkos::Impl::cuda_internal_safe_call (\r\n    e=cudaErrorIllegalAddress,\r\n---Type <return> to continue, or q <return> to quit---\r\n    name=0x1061cc70 <Kokkos::(anonymous namespace)::AllowPadding+58> \"cudaDeviceSynchronize()\",\r\n    file=0x1061cc38 <Kokkos::(anonymous namespace)::AllowPadding+2> \"../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp\", line=119)\r\n    at ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Error.hpp:58\r\n#9  0x0000000010596350 in Kokkos::Impl::cuda_device_synchronize ()\r\n    at ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:119\r\n#10 0x0000000010598b84 in Kokkos::Cuda::fence ()\r\n    at ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:824\r\n#11 0x0000000010051ea4 in Kokkos::Impl::CudaParallelLaunch<Kokkos::Impl::ParallelReduce<SPARTA_NS::FixAveHistoWeightKokkos, Kokkos::RangePolicy<SPARTA_NS::TagFixAveHistoWeight_BinGridCells2, Kokkos::Cuda>, Kokkos::MinMax<double, Kokkos::Cuda>, Kokkos::Cuda>, Kokkos::LaunchBounds<0u, 0u>, true>::CudaParallelLaunch (\r\n    this=0x7fffffff9161, driver=..., grid=..., block=..., shmem=4128,\r\n    stream=0x0) at ../../lib/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:303\r\n#12 0x000000001004d090 in Kokkos::Impl::ParallelReduce<SPARTA_NS::FixAveHistoWeightKokkos, Kokkos::RangePolicy<SPARTA_NS::TagFixAveHistoWeight_BinGridCells2, Kokkos::Cuda>, Kokkos::MinMax<double, Kokkos::Cuda>, Kokkos::Cuda>::execute (\r\n    this=0x7fffffff96a0)\r\n    at ../../lib/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp:925\r\n#13 0x0000000010045ef0 in Kokkos::Impl::ParallelReduceAdaptor<Kokkos::RangePolicy<SPARTA_NS::TagFixAveHistoWeight_BinGridCells2, Kokkos::Cuda>, SPARTA_NS::FixAveHistoWeightKokkos, Kokkos::MinMax<double, Kokkos::Cuda> >::execute (\r\n---Type <return> to continue, or q <return> to quit---\r\n    label=..., policy=..., functor=..., return_value=...)\r\n    at ../../lib/kokkos/core/src/Kokkos_Parallel_Reduce.hpp:889\r\n#14 0x00000000100418e4 in Kokkos::parallel_reduce<Kokkos::RangePolicy<SPARTA_NS::TagFixAveHistoWeight_BinGridCells2, Kokkos::Cuda>, SPARTA_NS::FixAveHistoWeightKokkos, Kokkos::MinMax<double, Kokkos::Cuda> > (policy=..., functor=...,\r\n    return_value=...)\r\n    at ../../lib/kokkos/core/src/Kokkos_Parallel_Reduce.hpp:972\r\n#15 0x000000001003a9c8 in SPARTA_NS::FixAveHistoWeightKokkos::bin_grid_cells (\r\n    this=0x1be96e00, reducer=..., d_vec=...)\r\n    at ../fix_ave_histo_weight_kokkos.cpp:420\r\n#16 0x00000000101481b8 in SPARTA_NS::FixAveHistoKokkos::end_of_step (\r\n    this=0x1be96e00) at ../fix_ave_histo_kokkos.cpp:296\r\n#17 0x0000000010147098 in SPARTA_NS::FixAveHistoKokkos::setup (this=0x1be96e00)\r\n    at ../fix_ave_histo_kokkos.cpp:126\r\n#18 0x00000000101251a0 in SPARTA_NS::Modify::setup (this=0x1bbc0e20)\r\n    at ../modify.cpp:132\r\n#19 0x000000001046d57c in SPARTA_NS::Update::setup (this=0x1bb80760)\r\n    at ../update.cpp:204\r\n#20 0x00000000104ea908 in SPARTA_NS::UpdateKokkos::setup (this=0x1bb80760)\r\n    at ../update_kokkos.cpp:240\r\n#21 0x00000000105158b8 in SPARTA_NS::Run::command (this=0x7fffffffa2e8,\r\n    narg=1, arg=0x1be5f6c0) at ../run.cpp:171\r\n#22 0x00000000102b2e6c in SPARTA_NS::Input::execute_command (this=0x1bb80480)\r\n---Type <return> to continue, or q <return> to quit---\r\n    at ../run.h:17\r\n#23 0x00000000102af720 in SPARTA_NS::Input::file (this=0x1bb80480)\r\n    at ../input.cpp:199\r\n#24 0x0000000010416c98 in main (argc=9, argv=0x7fffffffab58) at ../main.cpp:30\r\n```"}}
{"actor":{"display_login":"stanmoore1"}}