{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6060","repository_url":"https://api.github.com/repos/rethinkdb/rethinkdb","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6060/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6060/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6060/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/6060","id":171501399,"number":6060,"title":"Add `--fast` option to `rethinkdb restore`","user":{"login":"marshall007","id":877936,"avatar_url":"https://avatars.githubusercontent.com/u/877936?v=3","gravatar_id":"","url":"https://api.github.com/users/marshall007","html_url":"https://github.com/marshall007","followers_url":"https://api.github.com/users/marshall007/followers","following_url":"https://api.github.com/users/marshall007/following{/other_user}","gists_url":"https://api.github.com/users/marshall007/gists{/gist_id}","starred_url":"https://api.github.com/users/marshall007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marshall007/subscriptions","organizations_url":"https://api.github.com/users/marshall007/orgs","repos_url":"https://api.github.com/users/marshall007/repos","events_url":"https://api.github.com/users/marshall007/events{/privacy}","received_events_url":"https://api.github.com/users/marshall007/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"larkost","id":7431361,"avatar_url":"https://avatars.githubusercontent.com/u/7431361?v=3","gravatar_id":"","url":"https://api.github.com/users/larkost","html_url":"https://github.com/larkost","followers_url":"https://api.github.com/users/larkost/followers","following_url":"https://api.github.com/users/larkost/following{/other_user}","gists_url":"https://api.github.com/users/larkost/gists{/gist_id}","starred_url":"https://api.github.com/users/larkost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larkost/subscriptions","organizations_url":"https://api.github.com/users/larkost/orgs","repos_url":"https://api.github.com/users/larkost/repos","events_url":"https://api.github.com/users/larkost/events{/privacy}","received_events_url":"https://api.github.com/users/larkost/received_events","type":"User","site_admin":false},"assignees":[{"login":"larkost","id":7431361,"avatar_url":"https://avatars.githubusercontent.com/u/7431361?v=3","gravatar_id":"","url":"https://api.github.com/users/larkost","html_url":"https://github.com/larkost","followers_url":"https://api.github.com/users/larkost/followers","following_url":"https://api.github.com/users/larkost/following{/other_user}","gists_url":"https://api.github.com/users/larkost/gists{/gist_id}","starred_url":"https://api.github.com/users/larkost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larkost/subscriptions","organizations_url":"https://api.github.com/users/larkost/orgs","repos_url":"https://api.github.com/users/larkost/repos","events_url":"https://api.github.com/users/larkost/events{/privacy}","received_events_url":"https://api.github.com/users/larkost/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/123","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.4-polish","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/123/labels","id":1737269,"number":123,"title":"2.4-polish","description":"","creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":61,"closed_issues":16,"state":"open","created_at":"2016-04-28T19:25:10Z","updated_at":"2016-08-17T23:33:33Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2016-08-16T19:48:00Z","updated_at":"2016-08-17T23:36:50Z","closed_at":"2016-08-17T23:33:33Z","body":"In cases where you don't care about downtime or are provisioning a brand new cluster, it would be nice to have an option that follows best practices for getting the data restored as quickly as possible. From my experience, this means:\r\n\r\n1. create db and tables with appropriate shard/replica settings\r\n1. restore data with `--no-secondary-indices`\r\n1. restore secondary indices\r\n\r\nDepending on the size of the machine, restoring onto a single shard/replica first and then updating the table config and letting it backfill to other nodes as a second step can sometimes be even faster. Some testing probably needs to be done to verify which is actually faster in the general case or if a heuristic can be applied to determine which method we should use.\r\n\r\nIn the future, I would like to see better throughput overall when restoring, perhaps with a dedicated, lower-level protocol for streaming data in and/or serializing straight to disk. For now, having a flag that at least performs these tasks in the optimal order seems like a good start.\r\n\r\nRegardless, the current behavior of sharding + creating indices up front is definitely the slowest. It would be nice not having to remember all this and micromanage the restore process each time."},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/240581863","html_url":"https://github.com/rethinkdb/rethinkdb/issues/6060#issuecomment-240581863","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/6060","id":240581863,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T23:36:50Z","updated_at":"2016-08-17T23:36:50Z","body":"> I would appreciate if someone could comment on exactly why restoring onto a cluster isn't always faster than a single node, though.\r\n\r\nThat's surprising to me too. If all replicas are on the local machine that `rethinkdb restore` is connecting to, that makes sense because it reduces the CPU and network overhead required for re-routing the writes to the appropriate node. However if you use a single shard in a cluster, RethinkDB will still pick a random server for the shard of each table. It will not necessarily be on the server that the `restore` script connects to. So I wouldn't expect this to provide any performance benefit."}}
{"actor":{"display_login":"danielmewes"}}