{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/222","repository_url":"https://api.github.com/repos/Azure/azure-storage-cpp","labels_url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/222/labels{/name}","comments_url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/222/comments","events_url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/222/events","html_url":"https://github.com/Azure/azure-storage-cpp/issues/222","id":366416745,"node_id":"MDU6SXNzdWUzNjY0MTY3NDU=","number":222,"title":"Continous downloading of 64M or larger objects from a container causes core dump(s)","user":{"login":"yxiang92128","id":17994247,"node_id":"MDQ6VXNlcjE3OTk0MjQ3","avatar_url":"https://avatars0.githubusercontent.com/u/17994247?v=4","gravatar_id":"","url":"https://api.github.com/users/yxiang92128","html_url":"https://github.com/yxiang92128","followers_url":"https://api.github.com/users/yxiang92128/followers","following_url":"https://api.github.com/users/yxiang92128/following{/other_user}","gists_url":"https://api.github.com/users/yxiang92128/gists{/gist_id}","starred_url":"https://api.github.com/users/yxiang92128/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yxiang92128/subscriptions","organizations_url":"https://api.github.com/users/yxiang92128/orgs","repos_url":"https://api.github.com/users/yxiang92128/repos","events_url":"https://api.github.com/users/yxiang92128/events{/privacy}","received_events_url":"https://api.github.com/users/yxiang92128/received_events","type":"User","site_admin":false},"labels":[{"id":62313586,"node_id":"MDU6TGFiZWw2MjMxMzU4Ng==","url":"https://api.github.com/repos/Azure/azure-storage-cpp/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2018-10-03T16:11:52Z","updated_at":"2018-10-25T02:14:37Z","closed_at":null,"author_association":"NONE","body":"Please help to see what possibly could have gone wrong in the attached test code. \r\n[test.cpp.txt](https://github.com/Azure/azure-storage-cpp/files/2442780/test.cpp.txt)\r\n\r\n\r\nEnvironment/Configuration of the test case:\r\n1.\tUbuntu: Ubuntu 16.04.5 LTS (Dual-core Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz with 16G of RAM)\r\n2.\tAzure Storage C++ SDK 5.1\r\n3.\tBoost 1.68\r\n4.\tC++ REST SDK 2.10.3\r\n\r\nThe test case is very straightforward.  A container was pre-populated with 500 of the 64M objects containing UTF8 blobs.\r\nAnd a test program (see attached) goes into a loop and calling download_range_to_stream and fetch the 64M blob into a utf8 buffer. The parallelism factor was set to 1. But setting it to 2 tends to cause the core dump faster.\r\n\r\nThe crash(es) occurred during the loop randomly sometimes after just couple of downloads and other times after few hundreds.  They appeared to be within the C++ REST SDK code base. And most importantly, they occur consistently with every run.\r\n\r\nThere are apparently two core dump(s).\r\n\r\nThe first stack trace captured the following:\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1093 offset=0 size=67108864\r\nmodified on=131820260130000000 Fri, 21 Sep 2018 17:53:33 GMT\r\ndownload of TestObject.large.1093 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1094 offset=0 size=67108864\r\nmodified on=131820260170000000 Fri, 21 Sep 2018 17:53:37 GMT\r\ndownload of TestObject.large.1094 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1095 offset=0 size=67108864\r\nmodified on=131820260210000000 Fri, 21 Sep 2018 17:53:41 GMT\r\ndownload of TestObject.large.1095 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1096 offset=0 size=67108864\r\nmodified on=131820260240000000 Fri, 21 Sep 2018 17:53:44 GMT\r\ndownload of TestObject.large.1096 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1097 offset=0 size=67108864\r\nmodified on=131820260280000000 Fri, 21 Sep 2018 17:53:48 GMT\r\ndownload of TestObject.large.1097 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.1098 offset=0 size=67108864\r\nmodified on=131820260310000000 Fri, 21 Sep 2018 17:53:51 GMT\r\n\r\nThread 28 \"test_azure\" received signal SIGSEGV, Segmentation fault.\r\n[Switching to Thread 0x7fffe2178700 (LWP 27582)]\r\n0x00000000004cdf29 in __gnu_cxx::__exchange_and_add (__mem=0x5c, __val=-1) at /usr/include/c++/5/ext/atomicity.h:49\r\n49\t  { return __atomic_fetch_add(__mem, __val, __ATOMIC_ACQ_REL); }\r\n(gdb) bt\r\n#0  0x00000000004cdf29 in __gnu_cxx::__exchange_and_add (__mem=0x5c, __val=-1) at /usr/include/c++/5/ext/atomicity.h:49\r\n#1  0x00000000004cdfc0 in __gnu_cxx::__exchange_and_add_dispatch (__mem=0x5c, __val=-1) at /usr/include/c++/5/ext/atomicity.h:82\r\n#2  0x00000000004dc797 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x54)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:147\r\n#3  0x00000000004d893d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fffa8001848, __in_chrg=<optimized out>)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:659\r\n#4  0x00007ffff7760bae in std::__shared_ptr<web::http::client::details::asio_connection, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (\r\n    this=0x7fffa8001840, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925\r\n#5  0x00007ffff7760be6 in std::shared_ptr<web::http::client::details::asio_connection>::~shared_ptr (this=0x7fffa8001840,\r\n    __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr.h:93\r\n#6  0x00007ffff7786d59 in std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection> > (__pointer=0x7fffa8001840)\r\n    at /usr/include/c++/5/bits/stl_construct.h:93\r\n#7  0x00007ffff7783237 in std::_Destroy_aux<false>::__destroy<std::shared_ptr<web::http::client::details::asio_connection>*> (\r\n    __first=0x7fffa8001840, __last=0x7fffa8001850) at /usr/include/c++/5/bits/stl_construct.h:103\r\n#8  0x00007ffff777ec12 in std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection>*> (__first=0x7fffa8001840,\r\n    __last=0x7fffa8001850) at /usr/include/c++/5/bits/stl_construct.h:126\r\n#9  0x00007ffff7778a7d in std::_Destroy<std::shared_ptr<web::http::client::details::asio_connection>*, std::shared_ptr<web::http::client::details::asio_connection> > (__first=0x7fffa8001840, __last=0x7fffa8001850) at /usr/include/c++/5/bits/stl_construct.h:151\r\n#10 0x00007ffff7773fca in std::vector<std::shared_ptr<web::http::client::details::asio_connection>, std::allocator<std::shared_ptr<web::http::client::details::asio_connection> > >::_M_erase_at_end (this=0x7fffbc003548, __pos=0x7fffa8001840) at /usr/include/c++/5/bits/stl_vector.h:1438\r\n#11 0x00007ffff776f029 in std::vector<std::shared_ptr<web::http::client::details::asio_connection>, std::allocator<std::shared_ptr<web::http::client::details::asio_connection> > >::_M_erase (this=0x7fffbc003548, __first=std::shared_ptr (empty) 0x0, __last=std::shared_ptr (empty) 0x0)\r\n    at /usr/include/c++/5/bits/vector.tcc:160\r\n#12 0x00007ffff776ae16 in std::vector<std::shared_ptr<web::http::client::details::asio_connection>, std::allocator<std::shared_ptr<web::http::client::details::asio_connection> > >::erase (this=0x7fffbc003548, __first=std::shared_ptr (empty) 0x0, __last=std::shared_ptr (empty) 0x0)\r\n    at /usr/include/c++/5/bits/stl_vector.h:1177\r\n#13 0x00007ffff7760dc8 in web::http::client::details::connection_pool_stack::free_stale_connections (this=0x7fffbc003540)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/src/http/client/http_client_asio.cpp:378\r\n#14 0x00007ffff776135b in web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}::operator()(boost::system::error_code const&) const (\r\n    __closure=0x7fffe2177c00, ec=...) at /var/opt/td/home/yang/cpprestsdk/Release/src/http/client/http_client_asio.cpp:485\r\n#15 0x00007ffff778a737 in boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code>::operator()() (\r\n    this=0x7fffe2177c00) at /usr/local/include/boost/asio/detail/bind_handler.hpp:65\r\n#16 0x00007ffff7786e80 in boost::asio::asio_handler_invoke<boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code> >(boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code>&, ...) (function=...)\r\n    at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69\r\n#17 0x00007ffff778353b in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code>, {lambda(boost::system::error_code const&)#1}>(boost::asio::detail::binder1<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::system::error_code>&, {lambda(boost::system::error_code const&)#1}&) (function=..., context=...)\r\n    at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37\r\n#18 0x00007ffff777ef19 in boost::asio::detail::handler_work<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}, boost::asio::system_executor>::complete<boos---Type <return> to continue, or q <return> to quit---\r\nt::asio::detail::binder1<{lambda(boost::system::error_code const&)#1}, boost::system::error_code> >(boost::asio::detail::binder1<{lambda(boost::system::error_code const&)#1}, boost::system::error_code>&, {lambda(boost::system::error_code const&)#1}&) (this=0x7fffe2177bd6, function=...,\r\n    handler=...) at /usr/local/include/boost/asio/detail/handler_work.hpp:82\r\n#19 0x00007ffff777946d in boost::asio::detail::wait_handler<web::http::client::details::asio_connection_pool::start_epoch_interval(std::shared_ptr<web::http::client::details::asio_connection_pool> const&)::{lambda(boost::system::error_code const&)#1}>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (owner=0x790e10, base=0x7fffd40013d0)\r\n    at /usr/local/include/boost/asio/detail/wait_handler.hpp:72\r\n#20 0x00007ffff765edfe in boost::asio::detail::scheduler_operation::complete (this=0x7fffd40013d0, owner=0x790e10, ec=..., bytes_transferred=0)\r\n    at /usr/local/include/boost/asio/detail/scheduler_operation.hpp:40\r\n#21 0x00007ffff7662bde in boost::asio::detail::scheduler::do_run_one (this=0x790e10, lock=..., this_thread=..., ec=...)\r\n    at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:401\r\n#22 0x00007ffff7662569 in boost::asio::detail::scheduler::run (this=0x790e10, ec=...)\r\n    at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:154\r\n#23 0x00007ffff7663216 in boost::asio::io_context::run (this=0x7ffff7bb9148 <crossplat::threadpool::shared_instance()::s_shared+8>)\r\n    at /usr/local/include/boost/asio/impl/io_context.ipp:62\r\n#24 0x00007ffff775d0bf in (anonymous namespace)::threadpool_impl::thread_start (\r\n    arg=0x7ffff7bb9140 <crossplat::threadpool::shared_instance()::s_shared>)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/src/pplx/threadpool.cpp:63\r\n#25 0x00007ffff775cfcf in (anonymous namespace)::threadpool_impl::<lambda()>::operator()(void) const (__closure=0x793cb8)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/src/pplx/threadpool.cpp:45\r\n#26 0x00007ffff775d6f8 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::<lambda()> >::run(void) (\r\n    this=0x793cb0) at /usr/local/include/boost/asio/detail/posix_thread.hpp:86\r\n#27 0x00007ffff7662e73 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x793cb0)\r\n    at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:74\r\n#28 0x00007ffff7bc16ba in start_thread (arg=0x7fffe2178700) at pthread_create.c:333\r\n#29 0x00007ffff291941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109\r\n(gdb)\r\n\r\nI suspect that in in web::http::client::details::connection_pool_stack::free_stale_connections, there is a locking issue among threads where connections vector had been emptied out before it reaches the code in question. Although from the look of it, the “            std::lock_guard<std::mutex> lock(self.m_lock);” was applied and it is still questionable why it would occur.\r\nNevertheless I made a kludge where an extra check for the vector is made before calling the erase in free_stale_connections:\r\n\r\ndiff -r casablanca/Release/src/http/client/http_client_asio.cpp cpprestsdk/Release/src/http/client/http_client_asio.cpp\r\n378c378,379\r\n<         m_connections.erase(m_connections.begin(), m_connections.begin() + m_highWater);\r\n---\r\n>         if ( !m_connections.empty())\r\n>            m_connections.erase(m_connections.begin(), m_connections.begin() + m_highWater);\r\n\r\nAnd this allowed me to have gone past the first SEGV.\r\n\r\nThe second coredump that followed is more puzzling and I have to add one note to this because this same exact crash was initially discovered on SLES12SP3 and only on parallelism factor being set to 2 or more. I decided to move the experiment to Ubuntu because that is your supported platform and was able now to duplicate the issue even with parallelism factor set to 1.\r\n\r\nAnd there is the stack trace:\r\nstarting number for this thread is: 2000\r\ncreating azure client handle with connection \r\n[New Thread 0x7fffc7142700 (LWP 10834)]\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.2000 offset=0 size=67108864\r\ndownload of TestObject.large.2000 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.2001 offset=0 size=67108864\r\ndownload of TestObject.large.2001 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.2002 offset=0 size=67108864\r\ndownload of TestObject.large.2002 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.2003 offset=0 size=67108864\r\ndownload of TestObject.large.2003 succeeded\r\nFinding container: stresstest\r\ndownload_blob_operation container=stresstest obj=TestObject.large.2004 offset=0 size=67108864\r\ndownload of TestObject.large.2004 succeeded\r\nFinding container: stresstest\r\n\r\nThread 42 \"test_azure\" received signal SIGTRAP, Trace/breakpoint trap.\r\n[Switching to Thread 0x7fffdb16a700 (LWP 10794)]\r\n0x00007ffff7bcb269 in raise (sig=5) at ../sysdeps/unix/sysv/linux/pt-raise.c:35\r\n35\t../sysdeps/unix/sysv/linux/pt-raise.c: No such file or directory.\r\n(gdb) bt\r\n#0  0x00007ffff7bcb269 in raise (sig=5) at ../sysdeps/unix/sysv/linux/pt-raise.c:35\r\n#1  0x00000000004d1a44 in pplx::details::_ExceptionHolder::~_ExceptionHolder (this=0x7fffb001c330, __in_chrg=<optimized out>)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplxtasks.h:946\r\n#2  0x00000000004fa190 in __gnu_cxx::new_allocator<pplx::details::_ExceptionHolder>::destroy<pplx::details::_ExceptionHolder> (\r\n    this=0x7fffb001c330, __p=0x7fffb001c330) at /usr/include/c++/5/ext/new_allocator.h:124\r\n#3  0x00000000004f4dfb in std::allocator_traits<std::allocator<pplx::details::_ExceptionHolder> >::destroy<pplx::details::_ExceptionHolder>\r\n    (__a=..., __p=0x7fffb001c330) at /usr/include/c++/5/bits/alloc_traits.h:542\r\n#4  0x00000000004ec6bf in std::_Sp_counted_ptr_inplace<pplx::details::_ExceptionHolder, std::allocator<pplx::details::_ExceptionHolder>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7fffb001c320) at /usr/include/c++/5/bits/shared_ptr_base.h:531\r\n#5  0x00000000004dbd74 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fffb001c320)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:150\r\n#6  0x00000000004d7f31 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7db568, __in_chrg=<optimized out>)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:659\r\n#7  0x00000000004d205c in std::__shared_ptr<pplx::details::_ExceptionHolder, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7db560,\r\n    __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925\r\n#8  0x00000000004d2094 in std::shared_ptr<pplx::details::_ExceptionHolder>::~shared_ptr (this=0x7db560, __in_chrg=<optimized out>)\r\n    at /usr/include/c++/5/bits/shared_ptr.h:93\r\n#9  0x00000000004d22ce in pplx::details::_Task_impl_base::~_Task_impl_base (this=0x7db550, __in_chrg=<optimized out>)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplxtasks.h:1710\r\n#10 0x00000000004f483c in pplx::details::_Task_impl<unsigned char>::~_Task_impl (this=0x7db550, __in_chrg=<optimized out>)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplxtasks.h:2413\r\n#11 0x00000000004fa107 in __gnu_cxx::new_allocator<pplx::details::_Task_impl<unsigned char> >::destroy<pplx::details::_Task_impl<unsigned char> > (this=0x7db550, __p=0x7db550) at /usr/include/c++/5/ext/new_allocator.h:124\r\n#12 0x00000000004f4d3b in std::allocator_traits<std::allocator<pplx::details::_Task_impl<unsigned char> > >::destroy<pplx::details::_Task_impl<unsigned char> > (__a=..., __p=0x7db550) at /usr/include/c++/5/bits/alloc_traits.h:542\r\n#13 0x00000000004ec1b5 in std::_Sp_counted_ptr_inplace<pplx::details::_Task_impl<unsigned char>, std::allocator<pplx::details::_Task_impl<unsigned char> >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7db540) at /usr/include/c++/5/bits/shared_ptr_base.h:531\r\n#14 0x00000000004dbd74 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7db540)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:150\r\n#15 0x00000000004d7f31 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7db680, __in_chrg=<optimized out>)\r\n    at /usr/include/c++/5/bits/shared_ptr_base.h:659\r\n#16 0x00000000004d380a in std::__shared_ptr<pplx::details::_Task_impl<unsigned char>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (\r\n    this=0x7db678, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr_base.h:925\r\n#17 0x00000000004d3826 in std::shared_ptr<pplx::details::_Task_impl<unsigned char> >::~shared_ptr (this=0x7db678,\r\n    __in_chrg=<optimized out>) at /usr/include/c++/5/bits/shared_ptr.h:93\r\n#18 0x00007ffff4d30520 in pplx::details::_PPLTaskHandle<unsigned char, pplx::task<unsigned char>::_ContinuationTaskHandle<void, void, azure::storage::cloud_blob::download_range_to_stream_async(Concurrency::streams::ostream, utility::size64_t, utility::size64_t, const azure::storage::access_condition&, const azure::storage::blob_request_options&, azure::storage::operation_context)::<lambda(pplx::task<void>)>, std::integral_constant<bool, true>, pplx::details::_TypeSelectorAsyncTask>, pplx::details::_ContinuationTaskHandleBase>::~_PPLTaskHandle(void) (\r\n    this=0x7db660, __in_chrg=<optimized out>) at /usr/local/include/pplx/pplxtasks.h:1620\r\n#19 0x00007ffff4d389f2 in pplx::task<unsigned char>::_ContinuationTaskHandle<void, void, azure::storage::cloud_blob::download_range_to_stream_async(Concurrency::streams::ostream, utility::size64_t, utility::size64_t, const azure::storage::access_condition&, const azure::storage::blob_request_options&, azure::storage::operation_context)::<lambda(pplx::task<void>)>, std::integral_constant<bool, true>, pplx::details::_TypeSelectorAsyncTask>::~_ContinuationTaskHandle(void) (this=0x7db660, __in_chrg=<optimized out>)\r\n    at /usr/local/include/pplx/pplxtasks.h:3852\r\n#20 0x00007ffff4d38a6a in pplx::task<unsigned char>::_ContinuationTaskHandle<void, void, azure::storage::cloud_blob::download_range_to_stream_async(Concurrency::streams::ostream, utility::size64_t, utility::size64_t, const azure::storage::access_condition&, const azure::storage::blob_request_options&, azure::storage::operation_context)::<lambda(pplx::task<void>)>, std::integral_constant<bool, true>, pplx::details::_TypeSelectorAsyncTask>::~_ContinuationTaskHandle(void) (this=0x7db660, __in_chrg=<optimized out>)\r\n    at /usr/local/include/pplx/pplxtasks.h:3852\r\n#21 0x00000000004d8677 in pplx::details::_AutoDeleter<pplx::details::_TaskProcHandle>::~_AutoDeleter (this=0x7fffdb1699e0,\r\n    __in_chrg=<optimized out>) at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplx.h:99\r\n#22 0x00000000004d1446 in pplx::details::_TaskProcHandle::_RunChoreBridge (_Parameter=0x7db660)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplx.h:115\r\n#23 0x00007ffff7557895 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (\r\n---Type <return> to continue, or q <return> to quit---\r\n    this=0x7fffdb169c08, f=@0x7fffdb169c00: 0x4d13ec <pplx::details::_TaskProcHandle::_RunChoreBridge(void*)>, a=...)\r\n    at /usr/local/include/boost/bind/bind.hpp:259\r\n#24 0x00007ffff7557830 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (\r\n    this=0x7fffdb169c00) at /usr/local/include/boost/bind/bind.hpp:1294\r\n#25 0x00007ffff75577e2 in boost::asio::asio_handler_invoke<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > > (function=...) at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69\r\n#26 0x00007ffff7557777 in boost_asio_handler_invoke_helpers::invoke<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >, boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > > (function=..., context=...)\r\n    at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37\r\n#27 0x00007ffff75576f5 in boost::asio::detail::handler_work<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >, boost::asio::system_executor>::complete<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > > (this=0x7fffdb169bf6, function=..., handler=...) at /usr/local/include/boost/asio/detail/handler_work.hpp:82\r\n#28 0x00007ffff7557554 in boost::asio::detail::completion_handler<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::do_complete (owner=0x78fe10, base=0x7fffb4011d10) at /usr/local/include/boost/asio/detail/completion_handler.hpp:70\r\n#29 0x00007ffff745b0ae in boost::asio::detail::scheduler_operation::complete (this=0x7fffb4011d10, owner=0x78fe10, ec=...,\r\n    bytes_transferred=0) at /usr/local/include/boost/asio/detail/scheduler_operation.hpp:40\r\n#30 0x00007ffff745ee8e in boost::asio::detail::scheduler::do_run_one (this=0x78fe10, lock=..., this_thread=..., ec=...)\r\n    at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:401\r\n#31 0x00007ffff745e819 in boost::asio::detail::scheduler::run (this=0x78fe10, ec=...)\r\n    at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:154\r\n#32 0x00007ffff745f4c6 in boost::asio::io_context::run (this=0x7ffff79b5168 <crossplat::threadpool::shared_instance()::s_shared+8>)\r\n    at /usr/local/include/boost/asio/impl/io_context.ipp:62\r\n#33 0x00007ffff755936f in (anonymous namespace)::threadpool_impl::thread_start (\r\n    arg=0x7ffff79b5160 <crossplat::threadpool::shared_instance()::s_shared>)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/src/pplx/threadpool.cpp:63\r\n#34 0x00007ffff755927f in (anonymous namespace)::threadpool_impl::<lambda()>::operator()(void) const (__closure=0x794568)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/src/pplx/threadpool.cpp:45\r\n#35 0x00007ffff75599a8 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::<lambda()> >::run(void) (this=0x794560) at /usr/local/include/boost/asio/detail/posix_thread.hpp:86\r\n#36 0x00007ffff745f123 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x794560)\r\n    at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:74\r\n#37 0x00007ffff7bc16ba in start_thread (arg=0x7fffdb16a700) at pthread_create.c:333\r\n#38 0x00007ffff291941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109\r\n(gdb)\r\n\r\n\r\nThe stack frames involved showed a double occurrences of the destructor called for the same task handle ( this=0x7db660) for #19 and #20.   I suspect the AutoDeleter inside struct _TaskProcHandle::_RunChoreBridge invoked the destructor once and then the task handle was a shared ptr as it goes out of the scope, the destructor got invoked again. \r\n\r\nThus I attempted to comment ouf the AutoDeleter as a kludge and the test run does last longer however it quickly ran out of memory and died again. Apparently without the AutoDeleter there is a memory leak with the runaway task handles. So I think the crash with the double free might be a rare occurrence but still not sure how it would happen. See the stack below:\r\n\r\n#19 0x00007ffff4d389f2 in pplx::task<unsigned char>::_ContinuationTaskHandle<void, void, azure::storage::cloud_blob::download_range_to_stream_async(Concurrency::streams::ostream, utility::size64_t, utility::size64_t, const azure::storage::access_condition&, const azure::storage::blob_request_options&, azure::storage::operation_context)::<lambda(pplx::task<void>)>, std::integral_constant<bool, true>, pplx::details::_TypeSelectorAsyncTask>::~_ContinuationTaskHandle(void) (this=0x7db660, __in_chrg=<optimized out>)\r\n    at /usr/local/include/pplx/pplxtasks.h:3852\r\n#20 0x00007ffff4d38a6a in pplx::task<unsigned char>::_ContinuationTaskHandle<void, void, azure::storage::cloud_blob::download_range_to_stream_async(Concurrency::streams::ostream, utility::size64_t, utility::size64_t, const azure::storage::access_condition&, const azure::storage::blob_request_options&, azure::storage::operation_context)::<lambda(pplx::task<void>)>, std::integral_constant<bool, true>, pplx::details::_TypeSelectorAsyncTask>::~_ContinuationTaskHandle(void) (this=0x7db660, __in_chrg=<optimized out>)\r\n    at /usr/local/include/pplx/pplxtasks.h:3852\r\n#21 0x00000000004d8677 in pplx::details::_AutoDeleter<pplx::details::_TaskProcHandle>::~_AutoDeleter (this=0x7fffdb1699e0,\r\n    __in_chrg=<optimized out>) at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplx.h:99\r\n#22 0x00000000004d1446 in pplx::details::_TaskProcHandle::_RunChoreBridge (_Parameter=0x7db660)\r\n    at /var/opt/td/home/yang/cpprestsdk/Release/include/pplx/pplx.h:115\r\n\r\n\r\nI am hoping to get this information out to you and to see if anybody in Azure Storage C++ and REST SDK world offer to help out or hint to a possible direction for further investigation.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/comments/432889507","html_url":"https://github.com/Azure/azure-storage-cpp/issues/222#issuecomment-432889507","issue_url":"https://api.github.com/repos/Azure/azure-storage-cpp/issues/222","id":432889507,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMjg4OTUwNw==","user":{"login":"katmsft","id":21050104,"node_id":"MDQ6VXNlcjIxMDUwMTA0","avatar_url":"https://avatars0.githubusercontent.com/u/21050104?v=4","gravatar_id":"","url":"https://api.github.com/users/katmsft","html_url":"https://github.com/katmsft","followers_url":"https://api.github.com/users/katmsft/followers","following_url":"https://api.github.com/users/katmsft/following{/other_user}","gists_url":"https://api.github.com/users/katmsft/gists{/gist_id}","starred_url":"https://api.github.com/users/katmsft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/katmsft/subscriptions","organizations_url":"https://api.github.com/users/katmsft/orgs","repos_url":"https://api.github.com/users/katmsft/repos","events_url":"https://api.github.com/users/katmsft/events{/privacy}","received_events_url":"https://api.github.com/users/katmsft/received_events","type":"User","site_admin":false},"created_at":"2018-10-25T02:14:37Z","updated_at":"2018-10-25T02:14:37Z","author_association":"COLLABORATOR","body":"Hi,\r\n\r\n1. I made sure that the parallelism is 2. In fact, you can take a look at the code I tested for the issue in [this gist](https://gist.github.com/katmsft/3d51e3141b69465e12629fa709962a7e).\r\n2. The code is run both within the Azure VM AND on my desktop, which is outside the Azure DC. I hit 3000+ iterations without any core dump/crashes.\r\n\r\nI do not think 5.2.0 would solve the problem, since the code change is not related to the issue in any way. However the latest cpprestsdk's master branch has introduced a fix for the known crash issue, perhaps you could give it a try, if you have time.\r\n\r\nCan you drop me an email so I could share with you a test machine that I use to reproduce the issue, and perhaps you could try to reproduce on the machine?\r\n\r\nBest regards,\r\n\r\nTank"}}
{"actor":{"display_login":"katmsft"}}