{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/52647747","id":52647747,"diff_hunk":"@@ -0,0 +1,86 @@\n+- Feature Name: Raft consistency checker\n+- Status: draft\n+- Start Date: 2016-02-10\n+- Authors: Ben Darnell, Bram Grunier, Vivek Menezes\n+- RFC PR: (PR # after acceptance of initial draft)\n+- Cockroach Issue: #837\n+\n+# Summary\n+\n+An online system that checks that periodically checks that range replicas\n+are consistent; a snapshot for a range at a specific point in the raft log\n+ has the same sha on all replicas.\n+\n+# Motivation\n+\n+Consistency! Correctness at scale.\n+\n+# Detailed design\n+\n+The proposal is for each range replica leader to periodically run the\n+consistency checker. The consistency checker runs in three phases:\n+\n+1. All replicas agree on the snapshot at which a sha is to be computed.\n+2. A sha is computed on all replicas.\n+3. The leader shares its sha with all replicas; a replica computing a\n+different sha from the one supplied by the leader alerts and eutanizes\n+the node.\n+\n+The above is implemented by introducing two raft commands:\n+\n+1. CHECK_CONSISTENCY_BEGIN (BEGIN): A replica on receiving this command\n+immediately acknowledges it has executed it, and asynchronously computes the sha\n+for the current snapshot. It continues executing commands while it computes the\n+sha on the side. Once the sha is computed it compares the BEGIN command id along\n+with the computed sha to the one received from the leader in the second command\n+below.\n+2. CHECK_CONSISTENCY_CHECKSUM (CHECKSUM): This command contains the id for the\n+BEGIN command it is referring to, and the sha that it has computed at the BEGIN\n+command snapshot on the leader. On receiving this command the replica saves the\n+command id and sha supplied, and compares the sha against the one it eventually\n+computes. Under the rare circumstances that it sees a difference, it will alert\n+and euthanize the node.\n+\n+Each range replica has the ability to keep track of multiple sha computations\n+that are being computed or recently computed. The shas get GC-ed after about\n+30 minutes.\n+\n+The periodic consistency checker is run within a scanner that scans through all\n+the local replicas and runs the consistency checker for all leader replicas.\n+The scanner runs in a continuous loop with equal time spacing between\n+replicas such that it spans the entire periodicity interval.\n+\n+Noteworthy scenarios:\n+\n+1. The leader dies between BEGIN and CHECKSUM: The replicas, including the new\n+leader, will continue computing the sha. The new leader will send a CHECKSUM\n+command with the sha.\n+2. The leadership change occurs between BEGIN and CHECKSUM: same as 1\n+3. The leader dies and is restored (still as leader) between BEGIN and CHECK: The\n+new leader doesn’t compute the sha, and the replicas never receive the sha and\n+eventually GC their sha.\n+4. The leader dies after sending the CHECKSUM: Replicas will continue computing\n+the sha, and will compared it against the one supplied. If the replica\n+itself has been elected leader and has noted a different sha, it will log an\n+alert but will stubbornly remain alive.\n+5. A replica dies after receiving BEGIN and receives the CHECKSUM later when it\n+comes back up: Since it died it will not have the computed sha, and will ignore\n+the CHECKSUM command. This behavior is unaffected by whether the replica is\n+being restored from a snapshot or not.\n+6. A replica dies after receiving BEGIN and never receives the CHECKSUM:\n+Nothing.\n+7. A new replica is added between BEGIN and CHECKSUM: The new replica receives\n+the CHECKSUM and ignores it.\n+8. A replica is removed between BEGIN and CHECK: Nothing.\n+\n+# Drawbacks\n+\n+None.\n+\n+# Alternatives\n+\n+A consistency checker that runs offline, or only in tests.","path":"docs/RFCS/raft_consistency_checker.md","position":82,"original_position":82,"commit_id":"969873aaa53bfe040cb875a6f22ce17bb450d6d1","original_commit_id":"969873aaa53bfe040cb875a6f22ce17bb450d6d1","user":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"body":"This is a great idea.","created_at":"2016-02-11T18:51:14Z","updated_at":"2016-02-11T18:51:14Z","html_url":"https://github.com/cockroachdb/cockroach/pull/4317#discussion_r52647747","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/52647747"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/4317#discussion_r52647747"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317","id":58987661,"html_url":"https://github.com/cockroachdb/cockroach/pull/4317","diff_url":"https://github.com/cockroachdb/cockroach/pull/4317.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/4317.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4317","number":4317,"state":"open","locked":false,"title":"RFC: online raft consistency checker [ci skip]","user":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"body":"\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.svg\" height=\"40\" alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/cockroachdb/cockroach/4317)\n<!-- Reviewable:end -->\n","created_at":"2016-02-11T03:18:40Z","updated_at":"2016-02-11T18:51:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"87bbed528bc520de111892601b12ed62a1a20120","assignee":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4317/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/969873aaa53bfe040cb875a6f22ce17bb450d6d1","head":{"label":"vivekmenezes:vivek/rfc","ref":"vivek/rfc","sha":"969873aaa53bfe040cb875a6f22ce17bb450d6d1","user":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"repo":{"id":34078715,"name":"cockroach","full_name":"vivekmenezes/cockroach","owner":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vivekmenezes/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":true,"url":"https://api.github.com/repos/vivekmenezes/cockroach","forks_url":"https://api.github.com/repos/vivekmenezes/cockroach/forks","keys_url":"https://api.github.com/repos/vivekmenezes/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vivekmenezes/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vivekmenezes/cockroach/teams","hooks_url":"https://api.github.com/repos/vivekmenezes/cockroach/hooks","issue_events_url":"https://api.github.com/repos/vivekmenezes/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/vivekmenezes/cockroach/events","assignees_url":"https://api.github.com/repos/vivekmenezes/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/vivekmenezes/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/vivekmenezes/cockroach/tags","blobs_url":"https://api.github.com/repos/vivekmenezes/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vivekmenezes/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vivekmenezes/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/vivekmenezes/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vivekmenezes/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/vivekmenezes/cockroach/languages","stargazers_url":"https://api.github.com/repos/vivekmenezes/cockroach/stargazers","contributors_url":"https://api.github.com/repos/vivekmenezes/cockroach/contributors","subscribers_url":"https://api.github.com/repos/vivekmenezes/cockroach/subscribers","subscription_url":"https://api.github.com/repos/vivekmenezes/cockroach/subscription","commits_url":"https://api.github.com/repos/vivekmenezes/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/vivekmenezes/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/vivekmenezes/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/vivekmenezes/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/vivekmenezes/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/vivekmenezes/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vivekmenezes/cockroach/merges","archive_url":"https://api.github.com/repos/vivekmenezes/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vivekmenezes/cockroach/downloads","issues_url":"https://api.github.com/repos/vivekmenezes/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/vivekmenezes/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/vivekmenezes/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/vivekmenezes/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vivekmenezes/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/vivekmenezes/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/vivekmenezes/cockroach/deployments","created_at":"2015-04-16T20:25:11Z","updated_at":"2016-01-22T03:53:14Z","pushed_at":"2016-02-11T03:17:11Z","git_url":"git://github.com/vivekmenezes/cockroach.git","ssh_url":"git@github.com:vivekmenezes/cockroach.git","clone_url":"https://github.com/vivekmenezes/cockroach.git","svn_url":"https://github.com/vivekmenezes/cockroach","homepage":null,"size":57153,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"89910e9e70965d0a5a3631c088b44d2ca9170069","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Survivable, Strongly-Consistent SQL Database","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/cockroachdb/cockroach/deployments","created_at":"2014-02-06T00:18:47Z","updated_at":"2016-02-11T16:47:55Z","pushed_at":"2016-02-11T18:44:48Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":"http://www.cockroachlabs.com","size":66398,"stargazers_count":6175,"watchers_count":6175,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":559,"mirror_url":null,"open_issues_count":302,"forks":559,"open_issues":302,"watchers":6175,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/4317"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/4317"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/4317/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/4317/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/969873aaa53bfe040cb875a6f22ce17bb450d6d1"}}}}