{"action":"closed","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6033","repository_url":"https://api.github.com/repos/rethinkdb/rethinkdb","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6033/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6033/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6033/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/6033","id":169525860,"number":6033,"title":"One instance crashed in Rethinkdb 2.2.5 cluster, possible broken data","user":{"login":"ntquyen","id":1430058,"avatar_url":"https://avatars.githubusercontent.com/u/1430058?v=3","gravatar_id":"","url":"https://api.github.com/users/ntquyen","html_url":"https://github.com/ntquyen","followers_url":"https://api.github.com/users/ntquyen/followers","following_url":"https://api.github.com/users/ntquyen/following{/other_user}","gists_url":"https://api.github.com/users/ntquyen/gists{/gist_id}","starred_url":"https://api.github.com/users/ntquyen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ntquyen/subscriptions","organizations_url":"https://api.github.com/users/ntquyen/orgs","repos_url":"https://api.github.com/users/ntquyen/repos","events_url":"https://api.github.com/users/ntquyen/events{/privacy}","received_events_url":"https://api.github.com/users/ntquyen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/pr:high","name":"pr:high","color":"e10c02"},{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"closed","locked":false,"assignee":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"assignees":[{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/126","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.3.5","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/126/labels","id":1910578,"number":126,"title":"2.3.5","description":"","creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":33,"state":"open","created_at":"2016-07-28T21:38:16Z","updated_at":"2016-08-22T18:32:36Z","due_on":"2016-08-04T07:00:00Z","closed_at":null},"comments":8,"created_at":"2016-08-05T04:22:23Z","updated_at":"2016-08-22T18:32:36Z","closed_at":"2016-08-22T18:32:36Z","body":"I'm running rethinkdb 2.2.5 cluster of 3 instances:\r\n\r\n- OS: coreos 899.15.0 (Linux coreos-data-4 4.3.6-coreos #2 SMP Tue Apr 5 10:32:16 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux)\r\n- Docker 1.9.1\r\n\r\nOne instance in the cluster suddenly crashed and cannot be restarted, backtrace: \r\n\r\n```\r\nRunning rethinkdb 2.2.5~0jessie (GCC 4.9.2)...\r\nRunning on Linux 4.3.6-coreos x86_64\r\nLoading data from directory /data\r\nListening for intracluster connections on port 29015\r\nConnected to server \"4200b06bc82a_1yx\" 83a15f4a-a772-4649-b866-d537e5c0cfba\r\nConnected to proxy ee2e9ad4-3bd7-48a0-ba3f-2c9406ac3d02\r\nConnected to server \"02f81613b7c1_wdx\" 8138c225-e59c-4e5c-8f2e-93a5689aea47\r\nListening for client driver connections on port 28015\r\nListening for administrative HTTP connections on port 8080\r\nListening on addresses: 127.0.0.1, 10.2.44.2, ::1, fe80::42:aff:fe02:2c02%5\r\nServer ready, \"c1d17389be71_29y\" d5a8f7d7-33b2-4f60-9561-38d4a9d163e1\r\nA newer version of the RethinkDB server is available: 2.3.4. You can read the changelog at <https://github.com/rethinkdb/rethinkdb/releases>.\r\nVersion: rethinkdb 2.2.5~0jessie (GCC 4.9.2)\r\nerror: Error in src/rdb_protocol/serialize_datum.cc at line 814:\r\nerror: Guarantee failed: [datum_deserialize(&read_stream, &type) == archive_result_t::SUCCESS] Deserialization of datum type from buf failed with error archive_result_t::RANGE_ERROR.\r\nerror: Backtrace:\r\nerror: Fri Aug  5 03:52:04 2016\r\n\r\n       1 [0xa8b7f0]: backtrace_t::backtrace_t() at 0xa8b7f0 (rethinkdb)\r\n       2 [0xa8bb69]: format_backtrace(bool) at 0xa8bb69 (rethinkdb)\r\n       3 [0xd0135d]: report_fatal_error(char const*, int, char const*, ...) at 0xd0135d (rethinkdb)\r\n       4 [0x92f080]: ql::datum_deserialize_from_buf(shared_buf_ref_t<char> const&, unsigned long) at 0x92f080 (rethinkdb)\r\n       5 [0x92f656]: ql::datum_deserialize_pair_from_buf(shared_buf_ref_t<char> const&, unsigned long) at 0x92f656 (rethinkdb)\r\n       6 [0x8d3329]: ql::datum_t::unchecked_get_pair(unsigned long) const at 0x8d3329 (rethinkdb)\r\n       7 [0x8d60fb]: rethinkdb() [0x8d60fb] at 0x8d60fb ()\r\n       8 [0x8d61d0]: ql::datum_t::is_ptype() const at 0x8d61d0 (rethinkdb)\r\n       9 [0x87668b]: ql::obj_or_seq_op_impl_t::eval_impl_dereferenced(ql::term_t const*, ql::scope_env_t*, ql::args_t*, scoped_ptr_t<ql::val_t> const&, std::function<scoped_ptr_t<ql::val_t> ()>) const at 0x87668b (rethinkdb)\r\n       10 [0x87952f]: ql::bracket_term_t::eval_impl(ql::scope_env_t*, ql::args_t*, ql::eval_flags_t) const at 0x87952f (rethinkdb)\r\n       11 [0x7d0952]: ql::op_term_t::term_eval(ql::scope_env_t*, ql::eval_flags_t) const at 0x7d0952 (rethinkdb)\r\n       12 [0x8ea64a]: ql::runtime_term_t::eval_on_current_stack(ql::scope_env_t*, ql::eval_flags_t) const at 0x8ea64a (rethinkdb)\r\n       13 [0x8ea94c]: ql::runtime_term_t::eval(ql::scope_env_t*, ql::eval_flags_t) const at 0x8ea94c (rethinkdb)\r\n       14 [0x7cd19a]: ql::op_term_t::maybe_grouped_data(ql::scope_env_t*, ql::argvec_t*, ql::eval_flags_t, counted_t<ql::grouped_data_t>*, scoped_ptr_t<ql::val_t>*) const at 0x7cd19a (rethinkdb)\r\n       15 [0x7d030b]: ql::op_term_t::term_eval(ql::scope_env_t*, ql::eval_flags_t) const at 0x7d030b (rethinkdb)\r\n       16 [0x8ea64a]: ql::runtime_term_t::eval_on_current_stack(ql::scope_env_t*, ql::eval_flags_t) const at 0x8ea64a (rethinkdb)\r\n       17 [0x8ea94c]: ql::runtime_term_t::eval(ql::scope_env_t*, ql::eval_flags_t) const at 0x8ea94c (rethinkdb)\r\n       18 [0x80ddcb]: ql::reql_func_t::call(ql::env_t*, std::vector<ql::datum_t, std::allocator<ql::datum_t> > const&, ql::eval_flags_t) const at 0x80ddcb (rethinkdb)\r\n       19 [0x80d436]: ql::func_t::call(ql::env_t*, ql::datum_t, ql::eval_flags_t) const at 0x80d436 (rethinkdb)\r\n       20 [0x7b8383]: compute_keys(store_key_t const&, ql::datum_t, sindex_disk_info_t const&, std::vector<std::pair<store_key_t, ql::datum_t>, std::allocator<std::pair<store_key_t, ql::datum_t> > >*) at 0x7b8383 (rethinkdb)\r\n       21 [0x7b9fb3]: rdb_update_single_sindex(store_t*, store_t::sindex_access_t const*, deletion_context_t const*, rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*) at 0x7b9fb3 (rethinkdb)\r\n       22 [0x7c2559]: callable_action_instance_t<std::_Bind<void (*(store_t*, store_t::sindex_access_t*, deletion_context_t const*, rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*))(store_t*, store_t::sindex_access_t const*, deletion_context_t const*, rdb_modification_report_t const*, unsigned long*, auto_drainer_t::lock_t, cond_t*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*, std::vector<std::pair<ql::datum_t, boost::optional<unsigned long> >, std::allocator<std::pair<ql::datum_t, boost::optional<unsigned long> > > >*)> >::run_action() at 0x7c2559 (rethinkdb)\r\n       23 [0x99c958]: coro_t::run() at 0x99c958 (rethinkdb)\r\nerror: Exiting.\r\n```\r\nWhen the crash occurred, `systemd-coredum` took all the CPU and cannot be stopped unless I stop the rethinkdb container and restart the server. After the server restarted and `systemd-coredum` disabled, rethinkdb still cannot be started, same crash.\r\n\r\nIt is likely the data is somehow broken (Deserialization failed). I'm have 10 rethinkdb clusters running in production but this is the first time I got this issue.\r\n"}}
{"actor":{"display_login":"danielmewes"}}