{"action":"created","issue":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/971","repository_url":"https://api.github.com/repos/QubesOS/qubes-issues","labels_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/971/labels{/name}","comments_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/971/comments","events_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/971/events","html_url":"https://github.com/QubesOS/qubes-issues/issues/971","id":71009748,"number":971,"title":"Improve qvm-backup key derivation/management","user":{"login":"andrewdavidwong","id":10417133,"avatar_url":"https://avatars.githubusercontent.com/u/10417133?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewdavidwong","html_url":"https://github.com/andrewdavidwong","followers_url":"https://api.github.com/users/andrewdavidwong/followers","following_url":"https://api.github.com/users/andrewdavidwong/following{/other_user}","gists_url":"https://api.github.com/users/andrewdavidwong/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewdavidwong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewdavidwong/subscriptions","organizations_url":"https://api.github.com/users/andrewdavidwong/orgs","repos_url":"https://api.github.com/users/andrewdavidwong/repos","events_url":"https://api.github.com/users/andrewdavidwong/events{/privacy}","received_events_url":"https://api.github.com/users/andrewdavidwong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/C:%20core","name":"C: core","color":"cccccc"},{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/crypto","name":"crypto","color":"d4c5f9"},{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/help%20wanted","name":"help wanted","color":"fad8c7"},{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/P:%20major","name":"P: major","color":"fbca04"},{"url":"https://api.github.com/repos/QubesOS/qubes-issues/labels/security","name":"security","color":"000000"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/milestones/17","html_url":"https://github.com/QubesOS/qubes-issues/milestones/Release%204.0","labels_url":"https://api.github.com/repos/QubesOS/qubes-issues/milestones/17/labels","id":1083579,"number":17,"title":"Release 4.0","description":"* core-admin rewrite AKA core3","creator":{"login":"marmarek","id":726704,"avatar_url":"https://avatars.githubusercontent.com/u/726704?v=3","gravatar_id":"","url":"https://api.github.com/users/marmarek","html_url":"https://github.com/marmarek","followers_url":"https://api.github.com/users/marmarek/followers","following_url":"https://api.github.com/users/marmarek/following{/other_user}","gists_url":"https://api.github.com/users/marmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/marmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmarek/subscriptions","organizations_url":"https://api.github.com/users/marmarek/orgs","repos_url":"https://api.github.com/users/marmarek/repos","events_url":"https://api.github.com/users/marmarek/events{/privacy}","received_events_url":"https://api.github.com/users/marmarek/received_events","type":"User","site_admin":false},"open_issues":141,"closed_issues":38,"state":"open","created_at":"2015-04-25T20:32:26Z","updated_at":"2016-09-21T16:08:59Z","due_on":null,"closed_at":null},"comments":87,"created_at":"2015-04-26T04:37:16Z","updated_at":"2016-09-27T23:56:04Z","closed_at":null,"body":"See:\r\nhttps://groups.google.com/d/msg/qubes-devel/CZ7WRwLXcnk/u_rZPoVxL5IJ"},"comment":{"url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/comments/250033129","html_url":"https://github.com/QubesOS/qubes-issues/issues/971#issuecomment-250033129","issue_url":"https://api.github.com/repos/QubesOS/qubes-issues/issues/971","id":250033129,"user":{"login":"marmarek","id":726704,"avatar_url":"https://avatars.githubusercontent.com/u/726704?v=3","gravatar_id":"","url":"https://api.github.com/users/marmarek","html_url":"https://github.com/marmarek","followers_url":"https://api.github.com/users/marmarek/followers","following_url":"https://api.github.com/users/marmarek/following{/other_user}","gists_url":"https://api.github.com/users/marmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/marmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmarek/subscriptions","organizations_url":"https://api.github.com/users/marmarek/orgs","repos_url":"https://api.github.com/users/marmarek/repos","events_url":"https://api.github.com/users/marmarek/events{/privacy}","received_events_url":"https://api.github.com/users/marmarek/received_events","type":"User","site_admin":false},"created_at":"2016-09-27T23:56:04Z","updated_at":"2016-09-27T23:56:04Z","body":"> @marmarek the gains from sparse file storage aren't so much on the read (backup) side (though, unlike your short benchmark, the are quite big if you have rotational disks like I do). They are mostly on the write side. When your system has to write an enormous file to disk, and allocate those zeroes that should be unallocated, you end up spending a MONSTROUS amount of disk space and disk activity just to store those zeroes. On a system like Qubes, which relies on thin provisioned storage, getting rid of sparse file storage is a bad idea.\r\n\r\nThis isn't a problem. `dd conv=sparse` in the middle does the trick.\r\n\r\n> Maybe a purpose-built, short C or Go program, that reads from device files and writes tar format to its output, is the right thing to use here. It avoids using tar directly, it can detect rows of zeroes and output them as sparse blocks, and it isn't needed during restore (as you can use tar directly in that case). Those are my thoughts. What do you think?\r\n\r\nThis is what I'm currently exploring, as I've failed to find any other method (tried many tar implementations, other archive formats etc)."}}
{"actor":{"display_login":"marmarek"}}