{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7600","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7600/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7600/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7600/events","html_url":"https://github.com/cockroachdb/cockroach/issues/7600","id":163491835,"number":7600,"title":"storage: clobbering between uninitialized replica and RHS of a split","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"assignees":[{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2016-07-02T00:03:38Z","updated_at":"2016-07-02T03:24:50Z","closed_at":null,"body":"As discussed in #6991, but now also directly observed (on `master`) in\r\n\r\n`$ make stressrace PKG=./storage TESTS=TestStoreRangeDownReplicate STRESSFLAGS='-p 16 -maxfails 1 -stderr'`:\r\n\r\n```\r\nE160701 20:01:54.022089 raft/raft.go:925  [group 2] 3 state.commit 0 is out of range [10, 10]\r\npanic: [group 2] 3 state.commit 0 is out of range [10, 10]\r\n\r\ngoroutine 320 [running]:\r\npanic(0x5611dc0, 0xc820994420)\r\n\t/usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3ff\r\ngithub.com/cockroachdb/cockroach/storage.(*raftLogger).Panicf(0xc820994330, 0x5bc7f20, 0x2b, 0xc82099c5c0, 0x4, 0x4)\r\n\t/Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/storage/raft.go:117 +0x1cb\r\ngithub.com/coreos/etcd/raft.(*raft).loadState(0xc82095c340, 0x6, 0x1, 0x0, 0x0, 0x0, 0x0)\r\n\t/Users/tschottdorf/go/src/github.com/coreos/etcd/raft/raft.go:925 +0x3f6\r\ngithub.com/coreos/etcd/raft.newRaft(0xc820945b38, 0x30303c820945860)\r\n\t/Users/tschottdorf/go/src/github.com/coreos/etcd/raft/raft.go:225 +0xcd9\r\ngithub.com/coreos/etcd/raft.NewRawNode(0xc820945b38, 0x0, 0x0, 0x0, 0xc8209fa190, 0x0, 0x0)\r\n```\r\n\r\n#7598 should fix this."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/230080976","html_url":"https://github.com/cockroachdb/cockroach/issues/7600#issuecomment-230080976","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7600","id":230080976,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-07-02T03:24:50Z","updated_at":"2016-07-02T03:24:50Z","body":"bisect blames https://github.com/cockroachdb/cockroach/pull/7468 (fixing preemptive snapshots). With the above explanation (and my debugging efforts) I don't see it having a direct influence though it theoretically cause similar issues (#7598), but this repro'ed reliably with my patch set there (in a version which would actually write the HardState). I never saw a preemptive snapshot applied for range 2 (which is always the one which causes the issue).\r\n\r\nMy current theory is that working preemptive snapshots allows the first range to make it to more nodes quicker, which allows the split race to happen."}}
{"actor":{"display_login":"tschottdorf"}}