{"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/models/issues/1360","repository_url":"https://api.github.com/repos/PaddlePaddle/models","labels_url":"https://api.github.com/repos/PaddlePaddle/models/issues/1360/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/models/issues/1360/comments","events_url":"https://api.github.com/repos/PaddlePaddle/models/issues/1360/events","html_url":"https://github.com/PaddlePaddle/models/issues/1360","id":369978720,"node_id":"MDU6SXNzdWUzNjk5Nzg3MjA=","number":1360,"title":"Cannot load any more object with static TLS while executing crnn_ctc on multi-devices","user":{"login":"wanghaoshuang","id":7534971,"node_id":"MDQ6VXNlcjc1MzQ5NzE=","avatar_url":"https://avatars0.githubusercontent.com/u/7534971?v=4","gravatar_id":"","url":"https://api.github.com/users/wanghaoshuang","html_url":"https://github.com/wanghaoshuang","followers_url":"https://api.github.com/users/wanghaoshuang/followers","following_url":"https://api.github.com/users/wanghaoshuang/following{/other_user}","gists_url":"https://api.github.com/users/wanghaoshuang/gists{/gist_id}","starred_url":"https://api.github.com/users/wanghaoshuang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanghaoshuang/subscriptions","organizations_url":"https://api.github.com/users/wanghaoshuang/orgs","repos_url":"https://api.github.com/users/wanghaoshuang/repos","events_url":"https://api.github.com/users/wanghaoshuang/events{/privacy}","received_events_url":"https://api.github.com/users/wanghaoshuang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-15T02:47:32Z","updated_at":"2018-10-16T02:08:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Error log:\r\n\r\n```\r\n  File \"/home/users/***/lib/python2.7/site-packages/paddle/fluid/parallel_executor.py\", line 260, in run\r\n    self.executor.run(fetch_list, fetch_var_name)\r\npaddle.fluid.core.EnforceNotMet: Failed to find dynamic library: libwarpctc.so ( dlopen: cannot load any more object with static TLS )\r\n Please specify its path correctly using following ways:\r\n Method. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS.\r\n For instance, issue command: export LD_LIBRARY_PATH=...\r\n Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. at [/home/users/***/Paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:157]\r\nPaddlePaddle Call Stacks:\r\n0       0x7f635169b626p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486\r\n1       0x7f6352bc3e9ep paddle::platform::dynload::GetWarpCTCDsoHandle() + 1822\r\n2       0x7f63527b7149p void std::__once_call_impl<std::_Bind_simple<decltype (get_warpctc_version({parm#1}...)) paddle::platform::dynload::DynLoad__get_warpctc_version::operator()<>()::{lambda()#1} ()> >() + 9\r\n3       0x7f63bf6edbe0p pthread_once + 80\r\n4       0x7f63527be4a8p paddle::operators::WarpCTCFunctor<paddle::platform::CUDADeviceContext>::operator()(paddle::framework::ExecutionContext const&, float const*, float*, int const*, int const*, int const*, unsigned long, unsigned long, unsigned long, float*) + 136\r\n5       0x7f63527c0abbp paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const + 2763\r\n6       0x7f63527c2863p std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::WarpCTCKernel<paddle::platform::CUDADeviceContext, float> >::operator()(char const*, char const*) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&) + 35\r\n7       0x7f6352a3f04cp paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 492\r\n8       0x7f6352a3b7bcp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 252\r\n9       0x7f63529494c7p\r\n10      0x7f6352967100p\r\n11      0x7f6352966975p paddle::framework::details::OpHandleBase::RunAndRecordEvent(std::function<void ()> const&) + 805\r\n12      0x7f6352948f9fp paddle::framework::details::ComputationOpHandle::RunImpl() + 95\r\n13      0x7f6352967a05p paddle::framework::details::OpHandleBase::Run(bool) + 117\r\n14      0x7f635290626ap\r\n15      0x7f635176c263p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35\r\n16      0x7f635176bb17p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39\r\n17      0x7f63bf6edbe0p pthread_once + 80\r\n18      0x7f6352905242p\r\n19      0x7f635176d7f4p ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const + 404\r\n20      0x7f636b73c070p\r\n21      0x7f63bf6e8df3p\r\n22      0x7f63bed0d2cdp clone + 109\r\n```"},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/models/issues/comments/430075631","html_url":"https://github.com/PaddlePaddle/models/issues/1360#issuecomment-430075631","issue_url":"https://api.github.com/repos/PaddlePaddle/models/issues/1360","id":430075631,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDA3NTYzMQ==","user":{"login":"wanghaoshuang","id":7534971,"node_id":"MDQ6VXNlcjc1MzQ5NzE=","avatar_url":"https://avatars0.githubusercontent.com/u/7534971?v=4","gravatar_id":"","url":"https://api.github.com/users/wanghaoshuang","html_url":"https://github.com/wanghaoshuang","followers_url":"https://api.github.com/users/wanghaoshuang/followers","following_url":"https://api.github.com/users/wanghaoshuang/following{/other_user}","gists_url":"https://api.github.com/users/wanghaoshuang/gists{/gist_id}","starred_url":"https://api.github.com/users/wanghaoshuang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanghaoshuang/subscriptions","organizations_url":"https://api.github.com/users/wanghaoshuang/orgs","repos_url":"https://api.github.com/users/wanghaoshuang/repos","events_url":"https://api.github.com/users/wanghaoshuang/events{/privacy}","received_events_url":"https://api.github.com/users/wanghaoshuang/received_events","type":"User","site_admin":false},"created_at":"2018-10-16T02:08:37Z","updated_at":"2018-10-16T02:08:37Z","author_association":"CONTRIBUTOR","body":"一个解决方案是：编译的时候关闭mkldnn相关的选项，因为mkldnn相关的库也会占用大量static TLS"}}
{"actor":{"display_login":"wanghaoshuang"}}