{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1678","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1678/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1678/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1678/events","html_url":"https://github.com/scylladb/scylla/issues/1678","id":177833414,"number":1678,"title":"Scylla crashes during trace flushing","user":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-09-19T16:35:20Z","updated_at":"2016-09-20T05:21:32Z","closed_at":null,"body":"*Installation details*\r\nScylla version (or git commit hash): 7a140a3\r\nCluster size: 3-3\r\nOS: Fedora 23\r\n\r\nFound while trying to reproduce #1647 \r\n\r\n```\r\n(gdb) bt\r\n#0  std::deque<tracing::event_record, std::allocator<tracing::event_record> >::size (this=<optimized out>) at /usr/include/c++/5.3.1/bits/stl_deque.h:1264\r\n#1  tracing::one_session_records::size (this=<optimized out>) at ./tracing/tracing.hh:261\r\n#2  tracing::trace_keyspace_helper::<lambda()>::operator() (__closure=<optimized out>) at tracing/trace_keyspace_helper.cc:236\r\n#3  seastar::with_gate<tracing::trace_keyspace_helper::write_one_session_records(lw_shared_ptr<tracing::one_session_records>)::<lambda()> > (func=<optimized out>, g=...)\r\n    at ./seastar/core/gate.hh:120\r\n#4  tracing::trace_keyspace_helper::write_one_session_records (this=this@entry=0x6000000e7a80, records=...) at tracing/trace_keyspace_helper.cc:235\r\n#5  0x000000000129942e in tracing::trace_keyspace_helper::<lambda(std::deque<lw_shared_ptr<tracing::one_session_records> >::value_type&)>::operator() (one_session_records_ptr=..., \r\n    __closure=<synthetic pointer>) at tracing/trace_keyspace_helper.cc:259\r\n#6  std::for_each<std::_Deque_iterator<lw_shared_ptr<tracing::one_session_records>, lw_shared_ptr<tracing::one_session_records>&, lw_shared_ptr<tracing::one_session_records>*>, tracing::trace_keyspace_helper::write_records_bulk(tracing::records_bulk&)::<lambda(std::deque<lw_shared_ptr<tracing::one_session_records> >::value_type&)> > (__f=..., __last=..., __first=...)\r\n    at /usr/include/c++/5.3.1/bits/stl_algo.h:3767\r\n#7  tracing::trace_keyspace_helper::write_records_bulk (this=0x6000000e7a80, bulk=...) at tracing/trace_keyspace_helper.cc:260\r\n#8  0x0000000000793d17 in tracing::tracing::write_pending_records (this=0x600008ff3e00) at ./tracing/tracing.hh:402\r\n#9  tracing::tracing::write_maybe (this=0x600008ff3e00) at ./tracing/tracing.hh:435\r\n#10 tracing::trace_state::trace (this=0x600008b601c8, message=...) at tracing/trace_state.hh:466\r\n#11 0x0000000000c6dfec in tracing::trace_state::trace<gms::inet_address const&> (this=0x600008b601c8, \r\n    fmt=fmt@entry=0x184d280 \"read_mutation_data handling is done, sending a response to /{}\") at ./tracing/trace_state.hh:477\r\n#12 0x0000000000c4873b in tracing::trace<char const (&) [63], gms::inet_address&> (p=...) at ./tracing/trace_state.hh:579\r\n#13 service::storage_proxy::<lambda(const rpc::client_info&, query::read_command, query::partition_range)>::<lambda(const partition_range&, shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr&)>::<lambda()>::operator() (__closure=0x600008f16710) at service/storage_proxy.cc:3303\r\n#14 future<foreign_ptr<lw_shared_ptr<reconcilable_result> > >::finally_body<service::storage_proxy::init_messaging_service()::<lambda(const rpc::client_info&, query::read_command, query::partition_range)>::<lambda(const partition_range&, shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr&)> mutable::<lambda()>, false>::operator() (\r\n    result=<unknown type in /home/tgrabiec/.ccm/scylla-3-3/node3/bin/scylla, CU 0xbb7fce8, DIE 0xc10c765>, this=0x600008f16710) at /home/tgrabiec/src/scylla/seastar/core/future.hh:976\r\n#15 futurize<future<foreign_ptr<lw_shared_ptr<reconcilable_result> > > >::apply<future<foreign_ptr<lw_shared_ptr<reconcilable_result> > >::finally_body<service::storage_proxy::init_messaging_service()::<lambda(const rpc::client_info&, query::read_command, query::partition_range)>::<lambda(const partition_range&, shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr&)> mutable::<lambda()>, false>, future<foreign_ptr<lw_shared_ptr<reconcilable_result> > > > (func=<unknown type in /home/tgrabiec/.ccm/scylla-3-3/node3/bin/scylla, CU 0xbb7fce8, DIE 0xc10c734>)\r\n    at /home/tgrabiec/src/scylla/seastar/core/future.hh:1221\r\n#16 future<foreign_ptr<lw_shared_ptr<reconcilable_result> > >::<lambda(auto:5&&)>::operator()<future_state<foreign_ptr<lw_shared_ptr<reconcilable_result> > > > (\r\n    state=<unknown type in /home/tgrabiec/.ccm/scylla-3-3/node3/bin/scylla, CU 0xbb7fce8, DIE 0xbf787b6>, __closure=0x600008f166e0) at /home/tgrabiec/src/scylla/seastar/core/future.hh:886\r\n#17 continuation<future<T>::then_wrapped(Func&&) [with Func = future<foreign_ptr<lw_shared_ptr<reconcilable_result> > >::finally_body<service::storage_proxy::init_messaging_service()::<lambda(const rpc::client_info&, query::read_command, query::partition_range)>::<lambda(const partition_range&, shared_ptr<service::storage_proxy>&, tracing::trace_state_ptr&)> mutable::<lambda()>, false>; Result = future<foreign_ptr<lw_shared_ptr<reconcilable_result> > >; T = {foreign_ptr<lw_shared_ptr<reconcilable_result> >}]::<lambda(auto:5&&)>, foreign_ptr<lw_shared_ptr<reconcilable_result> > >::run(void) (this=0x600008f166c0) at /home/tgrabiec/src/scylla/seastar/core/future.hh:389\r\n#18 0x000000000048b92f in reactor::run_tasks (this=this@entry=0x6000002c2000, tasks=...) at core/reactor.cc:1896\r\n#19 0x00000000004c5efb in reactor::run (this=0x6000002c2000) at core/reactor.cc:2316\r\n#20 0x00000000005414b4 in app_template::run_deprecated(int, char**, std::function<void ()>&&) (this=this@entry=0x7ffe62003df0, ac=ac@entry=21, av=av@entry=0x7ffe62004068, \r\n    func=func@entry=<unknown type in /home/tgrabiec/.ccm/scylla-3-3/node3/bin/scylla, CU 0x74435e, DIE 0x80ec88>) at core/app-template.cc:131\r\n#21 0x000000000041da5b in main (ac=21, av=0x7ffe62004068) at main.cc:662\r\n```\r\n\r\nThe `records` pointer (the inner-most one) is nullptr:\r\n\r\n```\r\nvoid trace_keyspace_helper::write_one_session_records(lw_shared_ptr<one_session_records> records) {\r\n    with_gate(_pending_writes, [this, records = std::move(records)] {\r\n        auto num_records = records->size();\r\n```\r\n\r\n```\r\n(gdb) disassemble \r\nDump of assembler code for function tracing::trace_keyspace_helper::write_one_session_records(lw_shared_ptr<tracing::one_session_records>):\r\n   0x0000000001297b10 <+0>:\tpush   %r15\r\n   0x0000000001297b12 <+2>:\tpush   %r14\r\n   0x0000000001297b14 <+4>:\tpush   %r13\r\n   0x0000000001297b16 <+6>:\tpush   %r12\r\n   0x0000000001297b18 <+8>:\tpush   %rbp\r\n   0x0000000001297b19 <+9>:\tpush   %rbx\r\n   0x0000000001297b1a <+10>:\tsub    $0xe8,%rsp\r\n   0x0000000001297b21 <+17>:\tcmpb   $0x0,0x38(%rdi)\r\n   0x0000000001297b25 <+21>:\tmov    (%rsi),%r12\r\n   0x0000000001297b28 <+24>:\tmovq   $0x0,(%rsi)\r\n   0x0000000001297b2f <+31>:\tjne    0x1298f1a <tracing::trace_keyspace_helper::write_one_session_records(lw_shared_ptr<tracing::one_session_records>)+5130>\r\n=> 0x0000000001297b35 <+37>:\tmov    0x118(%r12),%rax\r\n```\r\n\r\n```\r\n(gdb) info registers \r\nrax            0x0\t0\r\nrbx            0x600008b16600\t105553262110208\r\nrcx            0x24\t36\r\nrdx            0x40\t64\r\nrsi            0x7ffe62003180\t140730542600576\r\nrdi            0x6000000e7a80\t105553117215360\r\nrbp            0x600008ff3e70\t0x600008ff3e70\r\nrsp            0x7ffe62003040\t0x7ffe62003040\r\nr8             0xffffffffffffffff\t-1\r\nr9             0xf000\t61440\r\nr10            0x0\t0\r\nr11            0x10\t16\r\nr12            0x0\t0\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/248204260","html_url":"https://github.com/scylladb/scylla/issues/1678#issuecomment-248204260","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1678","id":248204260,"user":{"login":"vladzcloudius","id":6103981,"avatar_url":"https://avatars.githubusercontent.com/u/6103981?v=3","gravatar_id":"","url":"https://api.github.com/users/vladzcloudius","html_url":"https://github.com/vladzcloudius","followers_url":"https://api.github.com/users/vladzcloudius/followers","following_url":"https://api.github.com/users/vladzcloudius/following{/other_user}","gists_url":"https://api.github.com/users/vladzcloudius/gists{/gist_id}","starred_url":"https://api.github.com/users/vladzcloudius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vladzcloudius/subscriptions","organizations_url":"https://api.github.com/users/vladzcloudius/orgs","repos_url":"https://api.github.com/users/vladzcloudius/repos","events_url":"https://api.github.com/users/vladzcloudius/events{/privacy}","received_events_url":"https://api.github.com/users/vladzcloudius/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T05:21:32Z","updated_at":"2016-09-20T05:21:32Z","body":"@duarten I described the main idea in the previous comment: one should\ncreate a secondary session on the non-local shard similarly to what we do\non a remote node.\n\nOn Sep 19, 2016 23:14, \"Duarte Nunes\" <notifications@github.com> wrote:\n\n> Oops, my bad. I'm seeing now trace_state doesn't support concurrent\n> access.\n>\n> It doesn't look like I can just copy it, because it holds a pointer to the\n> local tracing object. I also can't easily use the existing ctors.\n>\n> @vladzcloudius <https://github.com/vladzcloudius> What's the best\n> approach to trace something across shards?\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/scylladb/scylla/issues/1678#issuecomment-248111327>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AF0jraU5GITWkU8FcHXvRZWKUgUzOMBNks5qru0hgaJpZM4KAsaK>\n> .\n>\n"}}
{"actor":{"display_login":"vladzcloudius"}}