{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6929","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6929/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6929/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6929/events","html_url":"https://github.com/cockroachdb/cockroach/issues/6929","id":157054399,"number":6929,"title":"kv: controlling range leadership","user":{"login":"andreimatei","id":377201,"avatar_url":"https://avatars.githubusercontent.com/u/377201?v=3","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"andreimatei","id":377201,"avatar_url":"https://avatars.githubusercontent.com/u/377201?v=3","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2016-05-26T18:59:03Z","updated_at":"2016-05-31T20:45:54Z","closed_at":null,"body":"I started working on a `TestCluster` class that's supposed to wrap a configurable number of TestServers and let tests control how ranges are replicated and who the leader of each one is. A higher-level MultiTestContext. It's gonna be used for testing DistSQL in general and in particular this wrapper I'm building over over the LeaderCache and the RangeDescriptorCache to be used for enquiring about the leadership of key spans. And David has tried to use something like this in the past for running more realistic (\"multi-node\") benchmarks. Seems like a good thing, right? It's gonna be great.\r\n\r\nOne area when I quickly ran into problems is how controlling the leadership of ranges is gonna work (technically leadership in the sense of holder of the LeaderLease, not Raft leadership). I would like the `TestCluster` to let the client say that it wants node _n1_ to become the leader of range _r1_. And 1ms later, it wants _n2_ to become the leader of that range.\r\nSeems like we don't currently have a way for a node to release its `LeaderLease`, and then we also don't have a clear way for force a node to become a leader when there's no leader.\r\nBesides being necessary for testing, we can imagine that this capability would be useful in the real world when we start talking about collocating ranges (and their leadership).\r\nI've talked to Tobias a bit about the topic, and here's a plan I want to drop in the pool and see if it makes a splash.\r\n\r\n1) We introduce a `ShortenLease(t)` raft command. This command can only be proposed by the current lease holder, and has the effect that it moves up the expiration of the current lease to time *t*. _t_ is chosen as the current clock of the leader (which is higher than the timestamps from its read cache - i.e. the highest timestamp it already served a read at).\r\nIn between when the command is proposed and when it's applied, the leader replica is gonna remember this t and refuse to serve reads above it.\r\nThe refusal is done by returning a `NotLeaderError`, redirecting to the _nextLeader_. _nextLeader_ is a hint given to the leader replica when `ShortenLease` is proposed, indicating if there's another replica that we intend to make the leader after the current lease expires. This is so we clients following the redirect don't proceed to random node and force it to inadvertently become the leader.\r\nThis `ShortenLease` sounds like it will be also useful for draining a node, particularly since Tobi says our LeaderLeases are gonna become longer than the current 1s when they get... tied... to... raft ticks(?).\r\n\r\nWe now have a way to expire leases. We still need a way to (probabilistically) elect a leader of the puppeteer's choosing. \r\n\r\n2) We introduce a `GetOrElectLeader(replicaID)` RPC method, which returns the current leader if there's a valid leader lease, or tries to acquire the lease otherwise - through the usual `LeaderLease` raft command. This is to be used for asking a node to become leader. It's also to be used for `LeaderLease` cache population purposes, when you don't know who the leader is, and would also like to favor one particular node to become leader if there is none (this will be useful for SQL planning, when you need to figure out who a leader is but you also have a preference on the subject).\r\nAn alternative here is to somehow use the `DistSender` directly to send a read command for the range to the node that you want to elect leader. The downside is that some DistSender methods for talking directly to a particular replica would have to be exposed for testing, and that we'd be doing read without caring about the actual values, just for its side effect of electing a leader.\r\n\r\n\r\nSo with these two together, a test has the means to try to move leadership at will. It's all probabilistic - there's no guarantee that a rando doesn't grab the lease in between shortening the old lease and acquiring the new one on the desired target, but the test can keep trying until it gets what it wants.\r\n\r\nDoes this sound sane?\r\n\r\n@tschottdorf @bdarnell @cuongdo @RaduBerinde \r\n\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/222814899","html_url":"https://github.com/cockroachdb/cockroach/issues/6929#issuecomment-222814899","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6929","id":222814899,"user":{"login":"andreimatei","id":377201,"avatar_url":"https://avatars.githubusercontent.com/u/377201?v=3","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T20:45:54Z","updated_at":"2016-05-31T20:45:54Z","body":"> What is TransferLease? (This is the only time that name is used)\r\n\r\nSorry, there's no `TransferLease`. What I'm really doing is using `LeaderLease"}}
{"actor":{"display_login":"andreimatei"}}