{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6484","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6484/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6484/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6484/events","html_url":"https://github.com/cockroachdb/cockroach/pull/6484","id":152908084,"number":6484,"title":"RFC: Rebalancing V2","user":{"login":"BramGruneir","id":1614265,"avatar_url":"https://avatars.githubusercontent.com/u/1614265?v=3","gravatar_id":"","url":"https://api.github.com/users/BramGruneir","html_url":"https://github.com/BramGruneir","followers_url":"https://api.github.com/users/BramGruneir/followers","following_url":"https://api.github.com/users/BramGruneir/following{/other_user}","gists_url":"https://api.github.com/users/BramGruneir/gists{/gist_id}","starred_url":"https://api.github.com/users/BramGruneir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BramGruneir/subscriptions","organizations_url":"https://api.github.com/users/BramGruneir/orgs","repos_url":"https://api.github.com/users/BramGruneir/repos","events_url":"https://api.github.com/users/BramGruneir/events{/privacy}","received_events_url":"https://api.github.com/users/BramGruneir/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2016-05-04T00:59:56Z","updated_at":"2016-05-06T16:31:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/6484","html_url":"https://github.com/cockroachdb/cockroach/pull/6484","diff_url":"https://github.com/cockroachdb/cockroach/pull/6484.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/6484.patch"},"body":"[ci skip]\r\n\r\nRebalancing is the redistribution of replicas to optimize for a chosen set of heuristics. Currently, each range leader runs a distributed algorithm that spreads replicas as evenly as possible across the stores in a cluster. We artificially limit the rate at which each node may move replicas to avoid the excessive thrashing of replicas that results from making rebalancing decisions based on outdated information (gossiped StoreDescriptors that are up to a minute old).\r\n\r\nAs detailed later in this document, we’ve weighed decentralized against centralized allocation algorithms, as well as different heuristics for evaluating whether replicas are properly balanced. For V2 of our replica allocator, we are adding a replica reservation step to the distributed allocator and intelligently increasing the frequency at which we gossip StoreDescriptors. These modifications will significantly reduce the time required to rebalance small-to-medium-sized clusters while avoiding the waste of resources caused by excessive thrashing of replicas.\r\n\r\nWe’re specifically not addressing load-based rebalancing or heterogeneous stores/nodes in v2. Moreover, we are not addressing the potential for data unavailability when more than one node goes offline. These are important problems that will be addressed in v3 or later.\r\n\r\n<!-- Reviewable:start -->\r\n---\r\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"35\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/cockroachdb/cockroach/6484)\r\n<!-- Reviewable:end -->\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/217491724","html_url":"https://github.com/cockroachdb/cockroach/pull/6484#issuecomment-217491724","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6484","id":217491724,"user":{"login":"cuongdo","id":1564582,"avatar_url":"https://avatars.githubusercontent.com/u/1564582?v=3","gravatar_id":"","url":"https://api.github.com/users/cuongdo","html_url":"https://github.com/cuongdo","followers_url":"https://api.github.com/users/cuongdo/followers","following_url":"https://api.github.com/users/cuongdo/following{/other_user}","gists_url":"https://api.github.com/users/cuongdo/gists{/gist_id}","starred_url":"https://api.github.com/users/cuongdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cuongdo/subscriptions","organizations_url":"https://api.github.com/users/cuongdo/orgs","repos_url":"https://api.github.com/users/cuongdo/repos","events_url":"https://api.github.com/users/cuongdo/events{/privacy}","received_events_url":"https://api.github.com/users/cuongdo/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T16:31:50Z","updated_at":"2016-05-06T16:31:50Z","body":"\n\n\n\nReview status: all files reviewed at latest revision, 30 unresolved discussions.\n\n---\n\n*[docs/RFCS/rebalancing_v2.md, line 443 \\[r1\\]](https://reviewable.io:443/reviews/cockroachdb/cockroach/6484#-KH0VjColzzgp5bIYrgX:-KH5sBXuUbPmzSJbXWlz:-971920577) ([raw file](https://github.com/cockroachdb/cockroach/blob/83721ad85d91530db68c6000a07efa7d4f19ba76/docs/RFCS/rebalancing_v2.md#L443)):*\nI'm out right now and coming back to the office next Wednesday. There's been a lot of good discussion on this RFC, but to converge on a finalized RFC more quickly, feel free to talk in person before I get back.\n\n---\n\n*[docs/RFCS/rebalancing_v2.md, line 423 \\[r2\\]](https://reviewable.io:443/reviews/cockroachdb/cockroach/6484#-KH5Yh2EoisbL_XP_k_1:-KH5r9OanCEpmuaRWO7i:1761736081) ([raw file](https://github.com/cockroachdb/cockroach/blob/28ed01ceac1d2e99da2d6c3ddf1baf021457fa9c/docs/RFCS/rebalancing_v2.md#L423)):*\nI agree that 1 TB is too small, but that is part of the point. Even iterating through 15,000+ range descriptors regularly as part of the steady state of a Cockroach cluster isn't ideal. Consider the case where some of those range descriptors are on the other side of a slow, expensive link.\n\n---\n\n\n*Comments from [Reviewable](https://reviewable.io:443/reviews/cockroachdb/cockroach/6484)*\n<!-- Sent from Reviewable.io -->\n"}}