{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6284","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6284/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6284/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6284/events","html_url":"https://github.com/cockroachdb/cockroach/issues/6284","id":150918719,"number":6284,"title":"storage: Freeze/Unfreeze stop-the-world Raft migrations","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/leader-proposed-raft","name":"leader-proposed-raft","color":"fef2c0"}],"state":"open","locked":false,"assignee":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"milestone":null,"comments":15,"created_at":"2016-04-25T16:31:33Z","updated_at":"2016-05-11T23:23:02Z","closed_at":null,"body":"This is the first step of #6166.\r\n\r\nImplement the `Freeze`/`Unfreeze` Raft commands suggested by @bdarnell in\r\n\\#5985: everything proposed between `Freeze` and `Unfreeze` applies as a\r\nnon-retryable error (and in particular, does not change state).\r\n\r\nThis opens the following (Raft) migration path:\r\n\r\n* **apply** `Freeze` on all Replicas (`./cockroach drain --for-update`)\r\n* stop all nodes\r\n* replace the binary\r\n* start all nodes (which applies `Unfreeze` on all Replicas).\r\n\r\nSince the replaced binary will not have to apply Raft log entries proposed by\r\nthe previous binary (due to the active freeze), all is well.\r\n\r\nThis is implemented as a special drain mode (think `./cockroach drain\r\n--for-update`) which carries out the no-client, passive mode drain described in\r\n\\#6283 and then has one task left: making sure that all Replicas in the cluster\r\nhave the `Freeze` command applied (at which point it announces that to the\r\ncommand line and exits).\r\n\r\nThe user must execute the above command on all nodes in the cluster. Once all\r\nnodes have successfully carried it out, the nodes can be terminated normally\r\n(`./cockroach quit`).\r\n\r\nLeader leases are not acquired for `Freeze`/`Unfreeze` commands. Raft\r\nproposal forwarding will lead to proposing multiple copies of the command, but\r\nthat isn't an issue (a `Freeze` when frozen is a noop like all other commands;\r\nan `Unfreeze` when not frozen is by definition a no-op. And since any starting\r\nnode unfreezes all of its replicas, recovery from any situation is\r\nstraightforward).\r\n\r\nA process which boots up will unfreeze all ranges. This makes restarting the\r\ncluster after the update simple. It does introduce the potential for user\r\nerror: A user might restart a new binary before shutting down the old nodes,\r\nand that binary would then participate in the old (still running) Raft group.\r\nHowever, in light of the more sophisticated RFC #6166 which will allow proper\r\nonline migrations for many changes, this seems acceptable.\r\nWe can also safe-guard against this by adding the proposing binary's version\r\nnumber to `Unfreeze` which would cause those replicas running an old version to\r\npanic.\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/218618915","html_url":"https://github.com/cockroachdb/cockroach/issues/6284#issuecomment-218618915","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6284","id":218618915,"user":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"created_at":"2016-05-11T23:23:02Z","updated_at":"2016-05-11T23:23:02Z","body":"Potentially-destructive actions should for now only be in the CLI, not the UI - we don't have authentication worked out for the UI yet. "}}