{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/70437189","id":70437189,"diff_hunk":"@@ -2799,6 +2799,39 @@ func (r *Replica) changeReplicasTrigger(ctx context.Context, batch engine.Batch,\n \t\t\t}\n \t\t})\n \t}\n+\n+\t// For the first range, if we're adding a replica or removing a replica and\n+\t// this is not the replica being removed, then gossip that information\n+\t// immediately so that DistSender doesn't spin until the next\n+\t// `sentinelGossipInterval` tick while trying to hit the meta range.\n+\t//\n+\t// NB: It's possible that some node is coming up with stale first range\n+\t// information and this would cause it to gossip. This is thought to be\n+\t// sufficiently rare that we're not worrying about it for now.\n+\t//\n+\t// TODO(dan): Write a test for this (#7644).\n+\t// It should be possible to set up a three-node cluster, wait for initial\n+\t// replication of the first range, and then to disable the replication queue\n+\t// and downreplicate so that the first range leader disappears. Subsequently\n+\t// we should be able to verify that the first range gossip updates accordingly\n+\t// in a short amount of time (and once we have lease transfers, we can check\n+\t// more).\n+\tif r.RangeID == 1 && (change.ChangeType == roachpb.ADD_REPLICA ||\n+\t\tr.store.StoreID() != change.Replica.StoreID) {\n+\t\tbatch.Defer(func() {\n+\t\t\t// We need to run the gossip in an async task because gossiping requires\n+\t\t\t// the range lease and we'll deadlock if we try to acquire it will\n+\t\t\t// holding processRaftMu. Specifically, Replica.redirectOnOrAcquireLease\n+\t\t\t// blocks waiting for the lease acquisition to finish but it can't finish\n+\t\t\t// because we're not processing raft messages due to holding\n+\t\t\t// processRaftMu (and running on the processRaft goroutine).\n+\t\t\tr.store.Stopper().RunAsyncTask(func() {","path":"storage/replica_command.go","position":30,"original_position":30,"commit_id":"fbe4f8e6518d665ec0f6ee1f4b9ff4a558c3b463","original_commit_id":"fbe4f8e6518d665ec0f6ee1f4b9ff4a558c3b463","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"body":"I don't think that would work the way you imagine it to (but it probably does, or you wouldn't have PR'ed this, which makes me wonder what I'm missing - maybe that particular test we were looking at bumps the clock all the time). The nodes still have the (now defunct) lease, so they won't actually gossip as a result of this call as far as I can tell (because the lease for `Now()` is still held by the node just removed, and even if the nodes took into account that it had been removed, they still couldn't shorten the existing lease).\r\n\r\nSeems worth investigating the above. My feeling is that we won't really be able to do anything useful here except for\r\na) gossipping the first range unconditionally from all followers (paying with the occasional stale gossip which could also cause the 1min unavailability) or\r\nb) waiting until we've incorporated lease transfers into the rebalancing process (at which point the lease holder will never be equal to the replica which is being removed).\r\nc) leave as is, but clearly point out that this fixes only some unit test, not reality.","created_at":"2016-07-12T13:28:14Z","updated_at":"2016-07-12T13:28:14Z","html_url":"https://github.com/cockroachdb/cockroach/pull/7766#discussion_r70437189","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/70437189"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/7766#discussion_r70437189"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766","id":77111781,"html_url":"https://github.com/cockroachdb/cockroach/pull/7766","diff_url":"https://github.com/cockroachdb/cockroach/pull/7766.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/7766.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7766","number":7766,"state":"open","locked":false,"title":"storage: eagerly gossip first range in changeReplicasTrigger","user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"body":"Fixes #7297.\n\n<!-- Reviewable:start -->\n---\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/cockroachdb/cockroach/7766)\n<!-- Reviewable:end -->\n","created_at":"2016-07-12T12:34:22Z","updated_at":"2016-07-12T13:28:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"70c3c092547e3a84a578b777af1b1808f5fa0ca5","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/7766/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/fbe4f8e6518d665ec0f6ee1f4b9ff4a558c3b463","head":{"label":"petermattis:pmattis/gossip-first-range","ref":"pmattis/gossip-first-range","sha":"fbe4f8e6518d665ec0f6ee1f4b9ff4a558c3b463","user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"repo":{"id":49213713,"name":"cockroach","full_name":"petermattis/cockroach","owner":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/petermattis/cockroach","description":"A Scalable, Survivable, Strongly-Consistent SQL Database","fork":true,"url":"https://api.github.com/repos/petermattis/cockroach","forks_url":"https://api.github.com/repos/petermattis/cockroach/forks","keys_url":"https://api.github.com/repos/petermattis/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/petermattis/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/petermattis/cockroach/teams","hooks_url":"https://api.github.com/repos/petermattis/cockroach/hooks","issue_events_url":"https://api.github.com/repos/petermattis/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/petermattis/cockroach/events","assignees_url":"https://api.github.com/repos/petermattis/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/petermattis/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/petermattis/cockroach/tags","blobs_url":"https://api.github.com/repos/petermattis/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/petermattis/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/petermattis/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/petermattis/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/petermattis/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/petermattis/cockroach/languages","stargazers_url":"https://api.github.com/repos/petermattis/cockroach/stargazers","contributors_url":"https://api.github.com/repos/petermattis/cockroach/contributors","subscribers_url":"https://api.github.com/repos/petermattis/cockroach/subscribers","subscription_url":"https://api.github.com/repos/petermattis/cockroach/subscription","commits_url":"https://api.github.com/repos/petermattis/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/petermattis/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/petermattis/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/petermattis/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/petermattis/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/petermattis/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/petermattis/cockroach/merges","archive_url":"https://api.github.com/repos/petermattis/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/petermattis/cockroach/downloads","issues_url":"https://api.github.com/repos/petermattis/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/petermattis/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/petermattis/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/petermattis/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/petermattis/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/petermattis/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/petermattis/cockroach/deployments","created_at":"2016-01-07T15:46:02Z","updated_at":"2016-06-27T02:20:29Z","pushed_at":"2016-07-12T13:09:55Z","git_url":"git://github.com/petermattis/cockroach.git","ssh_url":"git@github.com:petermattis/cockroach.git","clone_url":"https://github.com/petermattis/cockroach.git","svn_url":"https://github.com/petermattis/cockroach","homepage":"http://www.cockroachlabs.com","size":146753,"stargazers_count":1,"watchers_count":1,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"b532139a6b62f02786823f1fc85b2beee78357b4","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Survivable, Strongly-Consistent SQL Database","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","deployments_url":"https://api.github.com/repos/cockroachdb/cockroach/deployments","created_at":"2014-02-06T00:18:47Z","updated_at":"2016-07-12T11:35:27Z","pushed_at":"2016-07-12T13:27:41Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":"https://www.cockroachlabs.com","size":214150,"stargazers_count":7237,"watchers_count":7237,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":708,"mirror_url":null,"open_issues_count":511,"forks":708,"open_issues":511,"watchers":7237,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/7766"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/7766"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/7766/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/7766/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/fbe4f8e6518d665ec0f6ee1f4b9ff4a558c3b463"}}}}
{"actor":{"display_login":"tschottdorf"}}