{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/4866","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/4866/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/4866/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/4866/events","html_url":"https://github.com/scylladb/scylla/issues/4866","id":482252410,"node_id":"MDU6SXNzdWU0ODIyNTI0MTA=","number":4866,"title":"Segmentation fault during multiple node hard reboot (MV-SI test)","user":{"login":"juliayakovlev","id":34435448,"node_id":"MDQ6VXNlcjM0NDM1NDQ4","avatar_url":"https://avatars1.githubusercontent.com/u/34435448?v=4","gravatar_id":"","url":"https://api.github.com/users/juliayakovlev","html_url":"https://github.com/juliayakovlev","followers_url":"https://api.github.com/users/juliayakovlev/followers","following_url":"https://api.github.com/users/juliayakovlev/following{/other_user}","gists_url":"https://api.github.com/users/juliayakovlev/gists{/gist_id}","starred_url":"https://api.github.com/users/juliayakovlev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juliayakovlev/subscriptions","organizations_url":"https://api.github.com/users/juliayakovlev/orgs","repos_url":"https://api.github.com/users/juliayakovlev/repos","events_url":"https://api.github.com/users/juliayakovlev/events{/privacy}","received_events_url":"https://api.github.com/users/juliayakovlev/received_events","type":"User","site_admin":false},"labels":[{"id":160570956,"node_id":"MDU6TGFiZWwxNjA1NzA5NTY=","url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true},{"id":463825172,"node_id":"MDU6TGFiZWw0NjM4MjUxNzI=","url":"https://api.github.com/repos/scylladb/scylla/labels/materialized-views","name":"materialized-views","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"assignees":[{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},{"login":"eliransin","id":22260843,"node_id":"MDQ6VXNlcjIyMjYwODQz","avatar_url":"https://avatars0.githubusercontent.com/u/22260843?v=4","gravatar_id":"","url":"https://api.github.com/users/eliransin","html_url":"https://github.com/eliransin","followers_url":"https://api.github.com/users/eliransin/followers","following_url":"https://api.github.com/users/eliransin/following{/other_user}","gists_url":"https://api.github.com/users/eliransin/gists{/gist_id}","starred_url":"https://api.github.com/users/eliransin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eliransin/subscriptions","organizations_url":"https://api.github.com/users/eliransin/orgs","repos_url":"https://api.github.com/users/eliransin/repos","events_url":"https://api.github.com/users/eliransin/events{/privacy}","received_events_url":"https://api.github.com/users/eliransin/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/31","html_url":"https://github.com/scylladb/scylla/milestone/31","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/31/labels","id":3968235,"node_id":"MDk6TWlsZXN0b25lMzk2ODIzNQ==","number":31,"title":"3.2","description":null,"creator":{"login":"slivne","id":3465480,"node_id":"MDQ6VXNlcjM0NjU0ODA=","avatar_url":"https://avatars0.githubusercontent.com/u/3465480?v=4","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"open_issues":149,"closed_issues":64,"state":"open","created_at":"2019-01-14T11:55:22Z","updated_at":"2019-08-20T05:04:56Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2019-08-19T11:18:24Z","updated_at":"2019-08-20T11:59:57Z","closed_at":null,"author_association":"NONE","body":"*Installation details*\r\nScylla version (or git commit hash): 3.1.0.rc3-0.20190816.d06bcef3b\r\nCluster size: 5 nodes\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): ami-07f2007dc543eced5\r\n\r\nCordump during MV-SI longevity test.\r\nSegmentation fault was received during multiple node hard reboot.\r\nThe reboot was on the same node. When rebooting was started 3th time, another node got coredump:\r\n\r\n```\r\n[centos@ip-172-30-0-181 ~]$ addr2line -Cpife /usr/lib/debug/opt/scylladb/libexec/scylla.bin-3.1.0.rc3-0.20190816.d06bcef3b.el7.x86_64.debug \r\n0x0000000004180932\r\n0x000000000406d9b5\r\n0x000000000406dcb5\r\n0x000000000406dd03\r\n0x00007f63ceb9202f\r\n0x0000000000536b17\r\n0x0000000001dc6b03\r\n0x0000000002e7546b\r\n0x0000000000837ec1\r\n```\r\n\r\nDecoded:\r\n```\r\nvoid seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/boost/program_options/variables_map.hpp:146\r\nseastar::backtrace_buffer::append_backtrace() at /usr/include/boost/program_options/variables_map.hpp:146\r\n (inlined by) print_with_backtrace at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/reactor.cc:1104\r\nseastar::print_with_backtrace(char const*) at /usr/include/boost/program_options/variables_map.hpp:146\r\nsigsegv_action at /usr/include/boost/program_options/variables_map.hpp:146\r\n (inlined by) operator() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/reactor.cc:5012\r\n (inlined by) _FUN at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/reactor.cc:5008\r\n?? ??:0\r\nstd::default_delete<seastar::internal::do_with_state<std::vector<frozen_mutation_and_schema, std::allocator<frozen_mutation_and_schema> >, seastar::future<> > >::operator()(seastar::internal::do_with_state<std::vector<frozen_mutation_and_schema, std::allocator<frozen_mutation_and_schema> >, seastar::future<> >*) const [clone .isra.3171] at storage_proxy.cc:?\r\nservice::storage_proxy::on_down(gms::inet_address const&) at crtstuff.c:?\r\n (inlined by) service::storage_proxy::on_down(gms::inet_address const&) at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/service/storage_proxy.cc:3636\r\noperator() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:768\r\n (inlined by) apply at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/apply.hh:35\r\n (inlined by) apply<service::storage_service::notify_down(service::storage_service::inet_address)::<lambda(auto:191&&)> [with auto:191 = service::storage_service&]::<lambda()> > at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/apply.hh:43\r\n (inlined by) apply_tuple<service::storage_service::notify_down(service::storage_service::inet_address)::<lambda(auto:191&&)> [with auto:191 = service::storage_service&]::<lambda()> > at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:1346\r\n (inlined by) apply<service::storage_service::notify_down(service::storage_service::inet_address)::<lambda(auto:191&&)> [with auto:191 = service::storage_service&]::<lambda()> > at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/future.hh:1380\r\n (inlined by) operator() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/core/thread.hh:324\r\n (inlined by) call at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/include/seastar/util/noncopyable_function.hh:71\r\nseastar::noncopyable_function<void ()>::operator()() const at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../include/seastar/util/noncopyable_function.hh:145\r\n (inlined by) seastar::thread_context::main() at /data/jenkins/workspace/scylla-3.1/relocatable-pkg/scylla/seastar/build/release/../../src/core/thread.cc:317\r\n```\r\n\r\nCoredump:\r\n```\r\n           PID: 2532 (scylla)\r\n           UID: 996 (scylla)\r\n           GID: 1001 (scylla)\r\n        Signal: 11 (SEGV)\r\n     Timestamp: Sun 2019-08-18 15:34:39 UTC (18h ago)\r\n  Command Line: /usr/bin/scylla /opt/scylladb/bin/../libexec/scylla.bin --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack \r\n    Executable: /opt/scylladb/libreloc/ld.so\r\n Control Group: /\r\n       Boot ID: 8bea027fb804493b870867b1a865f630\r\n    Machine ID: df877a200226bc47d06f26dae0736ec9\r\n      Hostname: ip-172-30-0-181.ec2.internal\r\n       Message: Process 2532 (scylla) of user 996 dumped core.\r\n                \r\n                Stack trace of thread 2548:\r\n                #0  0x0000000000536b18 _ZN7service13storage_proxy23remove_response_handlerEm (/opt/scylladb/libexec/scylla.bin)\r\n```\r\nFull coredump info was uploaded to ```upload.scylladb.com/core.scylla.996.8bea027fb804493b870867b1a865f630.2532.1566142479000000.gz/```\r\n\r\nThe cluster and node with coredump are alive now"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/522981550","html_url":"https://github.com/scylladb/scylla/issues/4866#issuecomment-522981550","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/4866","id":522981550,"node_id":"MDEyOklzc3VlQ29tbWVudDUyMjk4MTU1MA==","user":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2019-08-20T11:59:57Z","updated_at":"2019-08-20T11:59:57Z","author_association":"CONTRIBUTOR","body":"@bhalevy why do you think it's similar to #4386? the backtrace looks very different. But I didn't take a look at the core file yet."}}
{"actor":{"display_login":"nyh"}}