{"action":"opened","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3326","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3326/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3326/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3326/events","html_url":"https://github.com/scylladb/scylla/issues/3326","id":309541215,"number":3326,"title":"Populating materialized views hang the node","user":{"login":"glommer","id":331197,"avatar_url":"https://avatars3.githubusercontent.com/u/331197?v=4","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-28T21:37:32Z","updated_at":"2018-03-28T21:37:32Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Populating materialized views, one of the nodes prints a lot of large stalls (16s and up) , at some point preventing even a `systemctl restart`\r\n\r\nI decoded a couple and they look a bit like this:\r\n\r\n```\r\nseastar::backtrace_buffer::append_backtrace() at /usr/src/debug/scylla-666.development/seastar/core/reactor.cc:303\r\n (inlined by) print_with_backtrace at /usr/src/debug/scylla-666.development/seastar/core/reactor.cc:324\r\nseastar::reactor::block_notifier(int) at /usr/src/debug/scylla-666.development/seastar/core/reactor.cc:615\r\n?? ??:0\r\n?? ??:0\r\n_ZN5boost14multiprecision6numberINS0_8backends15cpp_int_backendILj0ELj0ELNS0_16cpp_integer_typeE1ELNS0_18cpp_int_check_typeE0ESaIyEEELNS0_26expression_template_optionE1EEC4EOS9_ at /opt/scylladb/include/boost/multiprecision/cpp_int.hpp:1076 (discriminator 7)\r\n (inlined by) _ZN9emptyableIN5boost14multiprecision6numberINS1_8backends15cpp_int_backendILj0ELj0ELNS1_16cpp_integer_typeE1ELNS1_18cpp_int_check_typeE0ESaIyEEELNS1_26expression_template_optionE1EEEEC4EOSA_ at /usr/src/debug/scylla-666.development/./types.hh:253 (discriminator 7)\r\n (inlined by) varint_type_impl::deserialize(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >) const at /usr/src/debug/scylla-666.development/types.cc:1375 (discriminator 7)\r\ndecimal_type_impl::deserialize(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >) const at /usr/src/debug/scylla-666.development/types.cc:1470\r\n (inlined by) decimal_type_impl::compare(std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >) const at /usr/src/debug/scylla-666.development/types.cc:1445\r\ntri_compare at /usr/src/debug/scylla-666.development/./types.hh:737\r\n (inlined by) int prefix_equality_tri_compare<__gnu_cxx::__normal_iterator<seastar::shared_ptr<abstract_type const> const*, std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > > >, compound_type<(allow_prefixes)1>::iterator, compound_type<(allow_prefixes)1>::iterator, int (*)(seastar::shared_ptr<abstract_type const>, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >)>(__gnu_cxx::__normal_iterator<seastar::shared_ptr<abstract_type const> const*, std::vector<seastar::shared_ptr<abstract_type const>, std::allocator<seastar::shared_ptr<abstract_type const> > > >, compound_type<(allow_prefixes)1>::iterator, compound_type<(allow_prefixes)1>::iterator, compound_type<(allow_prefixes)1>::iterator, compound_type<(allow_prefixes)1>::iterator, int (*)(seastar::shared_ptr<abstract_type const>, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >, std::experimental::fundamentals_v1::basic_string_view<signed char, std::char_traits<signed char> >)) at /usr/src/debug/scylla-666.development/./types.hh:187\r\n (inlined by) bound_view::tri_compare::operator()(clustering_key_prefix const&, int, clustering_key_prefix const&, int) const at /usr/src/debug/scylla-666.development/clustering_bounds_comparator.hh:73\r\n (inlined by) int position_in_partition::tri_compare::compare<position_in_partition_view, position_in_partition_view>(position_in_partition_view const&, position_in_partition_view const&) const at /usr/src/debug/scylla-666.development/position_in_partition.hh:394\r\n (inlined by) position_in_partition::tri_compare::operator()(position_in_partition_view const&, position_in_partition_view const&) const at /usr/src/debug/scylla-666.development/position_in_partition.hh:402\r\n (inlined by) rows_entry::tri_compare::operator()(position_in_partition_view, position_in_partition_view) const at /usr/src/debug/scylla-666.development/mutation_partition.hh:788\r\n (inlined by) rows_entry::compare::operator()(rows_entry const&, position_in_partition_view) const at /usr/src/debug/scylla-666.development/mutation_partition.hh:810\r\n (inlined by) bool intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare>::operator()<position_in_partition_view>(boost::intrusive::compact_rbtree_node<void*>* const&, position_in_partition_view const&) at /usr/src/debug/scylla-666.development/intrusive_set_external_comparator.hh:80\r\n (inlined by) boost::intrusive::compact_rbtree_node<void*>* boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, true> >::lower_bound_loop<position_in_partition_view, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare> >(boost::intrusive::compact_rbtree_node<void*>*, boost::intrusive::compact_rbtree_node<void*>*, position_in_partition_view const&, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare>) at /opt/scylladb/include/boost/intrusive/bstree_algorithms.hpp:2025\r\n (inlined by) boost::intrusive::compact_rbtree_node<void*>* boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, true> >::lower_bound<position_in_partition_view, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare> >(boost::intrusive::compact_rbtree_node<void*> const* const&, position_in_partition_view const&, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare>) at /opt/scylladb/include/boost/intrusive/bstree_algorithms.hpp:917\r\n (inlined by) boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, false> intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::lower_bound<position_in_partition_view, rows_entry::compare>(position_in_partition_view const&, rows_entry::compare) at /usr/src/debug/scylla-666.development/intrusive_set_external_comparator.hh:214\r\n (inlined by) partition_snapshot_row_cursor::prepare_heap(position_in_partition_view) at /usr/src/debug/scylla-666.development/partition_snapshot_row_cursor.hh:170\r\n (inlined by) partition_snapshot_row_cursor::ensure_entry_if_complete(position_in_partition_view) at /usr/src/debug/scylla-666.development/partition_snapshot_row_cursor.hh:350\r\npartition_entry::apply_to_incomplete(schema const&, partition_version*, logalloc::region&, cache_tracker&) at /usr/src/debug/scylla-666.development/partition_version.cc:466\r\noperator() at /usr/src/debug/scylla-666.development/partition_version.cc:428\r\n (inlined by) with_detached_versions<partition_entry::apply_to_incomplete(const schema&, partition_entry&&, const schema&, logalloc::region&, cache_tracker&)::<lambda(partition_version*)> > at /usr/src/debug/scylla-666.development/partition_version.cc:415\r\n (inlined by) partition_entry::apply_to_incomplete(schema const&, partition_entry&&, schema const&, logalloc::region&, cache_tracker&) at /usr/src/debug/scylla-666.development/partition_version.cc:427\r\noperator() at /usr/src/debug/scylla-666.development/row_cache.cc:1051\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/row_cache.cc:1016\r\n (inlined by) with_linearized_managed_bytes<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()>::<lambda()> mutable::<lambda()>::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/utils/managed_bytes.hh:423\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/row_cache.cc:1009\r\n (inlined by) with_reclaiming_disabled<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()>::<lambda()> mutable::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()>&> at /usr/src/debug/scylla-666.development/utils/logalloc.hh:710\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/utils/logalloc.hh:733\r\n (inlined by) with_reserve<logalloc::allocating_section::operator()(logalloc::region&, Func&&) [with Func = row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()>::<lambda()> mutable::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()>]::<lambda()> > at /usr/src/debug/scylla-666.development/utils/logalloc.hh:683\r\n (inlined by) operator()<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()>::<lambda()> mutable::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/utils/logalloc.hh:733\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/row_cache.cc:1001\r\n (inlined by) with_allocator<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()>::<lambda()> mutable [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/utils/allocation_strategy.hh:291\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/row_cache.cc:997\r\napply at /usr/src/debug/scylla-666.development/seastar/core/apply.hh:36\r\n (inlined by) apply<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/seastar/core/apply.hh:44\r\n (inlined by) do_void_futurize_apply_tuple<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/seastar/core/future.hh:1330\r\n (inlined by) apply<row_cache::do_update(row_cache::external_updater, memtable&, Updater)::<lambda()> [with Updater = row_cache::update(row_cache::external_updater, memtable&)::<lambda(boost::intrusive::set<cache_entry, boost::intrusive::member_hook<cache_entry, boost::intrusive::set_member_hook<boost::intrusive::link_mode<(boost::intrusive::link_mode_type)2> >, &cache_entry::_cache_link>, boost::intrusive::constant_time_size<false>, boost::intrusive::compare<cache_entry::compare> >::iterator, memtable_entry&, partition_presence_checker&)>]::<lambda()> > at /usr/src/debug/scylla-666.development/seastar/core/future.hh:1350\r\n (inlined by) operator() at /usr/src/debug/scylla-666.development/seastar/core/thread.hh:317\r\n (inlined by) _M_invoke at /opt/scylladb/include/c++/7/bits/std_function.h:316\r\nstd::function<void ()>::operator()() const at /opt/scylladb/include/c++/7/bits/std_function.h:706\r\n (inlined by) seastar::thread_context::main() at /usr/src/debug/scylla-666.development/seastar/core/thread.cc:310\r\n?? ??:0\r\n\r\n```\r\n\r\n\r\n"}}
{"actor":{"display_login":"glommer"}}