{"action":"created","issue":{"url":"https://api.github.com/repos/martinsumner/leveled/issues/164","repository_url":"https://api.github.com/repos/martinsumner/leveled","labels_url":"https://api.github.com/repos/martinsumner/leveled/issues/164/labels{/name}","comments_url":"https://api.github.com/repos/martinsumner/leveled/issues/164/comments","events_url":"https://api.github.com/repos/martinsumner/leveled/issues/164/events","html_url":"https://github.com/martinsumner/leveled/issues/164","id":357236232,"node_id":"MDU6SXNzdWUzNTcyMzYyMzI=","number":164,"title":"Hot backup","user":{"login":"martinsumner","id":1628897,"node_id":"MDQ6VXNlcjE2Mjg4OTc=","avatar_url":"https://avatars0.githubusercontent.com/u/1628897?v=4","gravatar_id":"","url":"https://api.github.com/users/martinsumner","html_url":"https://github.com/martinsumner","followers_url":"https://api.github.com/users/martinsumner/followers","following_url":"https://api.github.com/users/martinsumner/following{/other_user}","gists_url":"https://api.github.com/users/martinsumner/gists{/gist_id}","starred_url":"https://api.github.com/users/martinsumner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinsumner/subscriptions","organizations_url":"https://api.github.com/users/martinsumner/orgs","repos_url":"https://api.github.com/users/martinsumner/repos","events_url":"https://api.github.com/users/martinsumner/events{/privacy}","received_events_url":"https://api.github.com/users/martinsumner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-05T13:43:27Z","updated_at":"2018-09-14T09:07:23Z","closed_at":null,"author_association":"OWNER","body":"Would like to have a simple equivalent to [mv-hot-backup](https://github.com/basho/leveldb/wiki/mv-hot-backup).\r\n\r\nThe leveldb hot backup is problematic due to write amplification.  On some Riak installations we see write amplification of 100 times (e.g. the on-disk change in GB is 100 times larger than the volume of alterations received in any given time window).  \r\n\r\nLeveled should ease this pain significantly, as typically write amplification is reduced by at least one order of magnitude.  Would be nice to have a Leveled backup solution that wasn't exposed to any risk of corruption.\r\n\r\nProposal is to have a book_backup(Pid) which will:\r\n\r\n- Prompt the journal to roll the active cdb file;\r\n- Take a snapshot of the Inker;\r\n- Return {async, BackupFn}.\r\n\r\nCalling BackupFun(BackupFolderName), will:\r\n\r\n- Empty the contents of the BackupFolderName;\r\n- Take the Inker manifest file (ignoring the new active file);\r\n- Create a hard linked file to each file in the manifest in the BackupFolderName (in a journal sub-directory);\r\n- Update each file reference in the manifest with the linked file in the BackupFolder;\r\n- Write the updated manifest to the BackupFolderName (in a journal sub-directory);\r\n- Close the snapshot.\r\n\r\nIt should then be possible to start a new Bookie from the BackupFolderName in the future, and have the ledger recreated.  \r\n\r\nThe backup folder can then be rsync'd off to an offline backup.  the rsync should represent just the changes in the Journal file (and not an amplified churn associated with the LSM tree).\r\n\r\nDue to [hard-linking](https://medium.com/@wendymayorgasegura/what-is-the-difference-between-a-hard-link-and-a-symbolic-link-8c0493041b62), the backup will not consume extra space, but will begin to consume more space as any of those Journal files are deleted due to compaction.\r\n"},"comment":{"url":"https://api.github.com/repos/martinsumner/leveled/issues/comments/421284633","html_url":"https://github.com/martinsumner/leveled/issues/164#issuecomment-421284633","issue_url":"https://api.github.com/repos/martinsumner/leveled/issues/164","id":421284633,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMTI4NDYzMw==","user":{"login":"martinsumner","id":1628897,"node_id":"MDQ6VXNlcjE2Mjg4OTc=","avatar_url":"https://avatars0.githubusercontent.com/u/1628897?v=4","gravatar_id":"","url":"https://api.github.com/users/martinsumner","html_url":"https://github.com/martinsumner","followers_url":"https://api.github.com/users/martinsumner/followers","following_url":"https://api.github.com/users/martinsumner/following{/other_user}","gists_url":"https://api.github.com/users/martinsumner/gists{/gist_id}","starred_url":"https://api.github.com/users/martinsumner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinsumner/subscriptions","organizations_url":"https://api.github.com/users/martinsumner/orgs","repos_url":"https://api.github.com/users/martinsumner/repos","events_url":"https://api.github.com/users/martinsumner/events{/privacy}","received_events_url":"https://api.github.com/users/martinsumner/received_events","type":"User","site_admin":false},"created_at":"2018-09-14T09:07:23Z","updated_at":"2018-09-14T09:07:23Z","author_association":"OWNER","body":"https://github.com/martinsumner/leveled/pull/168\r\n\r\nThere are some implementation changes to the above:\r\n\r\nThe BackupFolder is not emptied, the contents are overwritten i.e. Backing up twice to the same folder, the second backup will update not over-write the previous abckup.  If a file already exists, and is required by the new backup - that file is not changed (as cdb files are immutable in leveled - it must still be the same).  If a file exists, but is not required in the new manifest, then it is removed.  "}}
{"actor":{"display_login":"martinsumner"}}