{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8944","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8944/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8944/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8944/events","html_url":"https://github.com/cockroachdb/cockroach/pull/8944","id":174039623,"number":8944,"title":"storage: eager GC for replicas blocking snapshots","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-30T14:46:27Z","updated_at":"2016-08-30T15:07:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/8944","html_url":"https://github.com/cockroachdb/cockroach/pull/8944","diff_url":"https://github.com/cockroachdb/cockroach/pull/8944.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/8944.patch"},"body":"Consider the following situation on a Range X:\r\n\r\n- the range lives on nodes 1,2,3\r\n- gets rebalanced to nodes 1,2,4 while node 3 is down\r\n- node 3 restarts, perhaps even leaving the (now removed) replica remains quiescent\r\n- the range splits (on 1, 2, and 4)\r\n- the right hand side of the split gets rebalanced to nodes 1, 2, 3.\r\n\r\nIn order to receive a snapshot for in the last step, node 3 needs to have\r\ngarbage-collected its old pre-split replica. If it weren't for node 3's\r\ndowntime this would normally happen eagerly as its peers will inform it of its\r\nfate. In this scenario however, one would have to wait until a) a client\r\nrequest creates the Raft group (which is unlikely as it isn't being addressed\r\nany more) or b) a queue picks it up (which can take a long time).\r\n\r\nInstead, when returning an overlap and the overlap appears to be inactive, we\r\nadd to the GC queue (which in turn activates the replica). For the situation\r\nabove, we could also hope to only create the Raft group (as that would likely,\r\nbut not necessarily, bring it into contact with its ex-peers). However, none of\r\nthe old members may still be around in larger clusters, so going for the GC\r\nqueue directly is the better option.\r\n\r\ncc @bdarnell - this would need a test and all, but it should address #8942 (without the part that moved to #8111).\n\n<!-- Reviewable:start -->\n---\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/cockroachdb/cockroach/8944)\n<!-- Reviewable:end -->\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/243471637","html_url":"https://github.com/cockroachdb/cockroach/pull/8944#issuecomment-243471637","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8944","id":243471637,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T15:07:25Z","updated_at":"2016-08-30T15:07:25Z","body":"Maybe you're thinking of the wrong issue? This is for your comment here:\r\n\r\n> In the second line, the replica was not re-added and so it should be GC'd. This is probably just a case of the replica GC queue being slow. We don't have a good way to do a local shouldQueue for replica GC so it ends up being inefficient.\r\n\r\nto allow the GC queue to get rid of what it can eagerly. The regular eager path (getting told by peers) doesn't work when Raft groups don't receive traffic (and are hence quiescent).\r\n\r\nIn turn, the GC queue itself is notoriously slow at finding replicas to GC on its own."}}
{"actor":{"display_login":"tschottdorf"}}