{"action":"closed","number":1601,"pull_request":{"url":"https://api.github.com/repos/duplicati/duplicati/pulls/1601","id":61807586,"html_url":"https://github.com/duplicati/duplicati/pull/1601","diff_url":"https://github.com/duplicati/duplicati/pull/1601.diff","patch_url":"https://github.com/duplicati/duplicati/pull/1601.patch","issue_url":"https://api.github.com/repos/duplicati/duplicati/issues/1601","number":1601,"state":"closed","locked":false,"title":"Reorganized `ScanForExistingTargetBlocks`","user":{"login":"FootStark","id":17236227,"avatar_url":"https://avatars.githubusercontent.com/u/17236227?v=3","gravatar_id":"","url":"https://api.github.com/users/FootStark","html_url":"https://github.com/FootStark","followers_url":"https://api.github.com/users/FootStark/followers","following_url":"https://api.github.com/users/FootStark/following{/other_user}","gists_url":"https://api.github.com/users/FootStark/gists{/gist_id}","starred_url":"https://api.github.com/users/FootStark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FootStark/subscriptions","organizations_url":"https://api.github.com/users/FootStark/orgs","repos_url":"https://api.github.com/users/FootStark/repos","events_url":"https://api.github.com/users/FootStark/events{/privacy}","received_events_url":"https://api.github.com/users/FootStark/received_events","type":"User","site_admin":false},"body":"This change tackles several challenges:\r\n1. It fixes a bug where duplicati truncates forthwritten files backuped\r\nwith a size being a multiple of blocksize instead of renaming them.\r\n2. It tries to verify file data on beginning when scanning for target blocks. If successful, marks files to\r\nnot be reverified in the end. This helps to speed up restore-into-existing scenarios.\r\n3. On overwrite restore, removes readonly attribute from files having to\r\nbe modified. Also only uses read mode, allowing scanning on Read-only files. This does not include necessary metadata changes, as they are not detected in `ScanForExistingTargetBlocks`. If this is correct actually depends on the overall strategy for how to handle Read-Only files on restore., which has to be discussed. When applying metadata, the attribute would be reset.\r\n4. A speedup by computing block and file Hashes in parallel when scanning.\r\n\r\nTo accomplish this, I added a Field `\"DataVerified\"` to the temp file tables. This can also be used when incorporating verification into the block restore procedure.\r\nI had to put the function `SetFileDataVerified`to mark files in the BlockMarker because of the Transaction, so it is actually now a FileAndBlockMarker.\r\nI decided to adjust file sizes already when scanning, not in the very end after restore (where it actually takes up a pretty long time). The downside of this is: If the backup is not available later on, all files larger than in the backup have already been truncated. But I do not think that is a problem: If someone decides to restore the files, he should not want the old ones around anyway.\r\n\r\nAs this change goes a little deeper into the whole restore process, you might verify thoroughly, but my tests have been successful yet. I hope you agree on the read-only attribute handling.\r\n","created_at":"2016-03-05T09:52:59Z","updated_at":"2016-03-06T23:09:05Z","closed_at":"2016-03-06T23:09:05Z","merged_at":"2016-03-06T23:09:05Z","merge_commit_sha":"9c7c70d497fc7d1531cec798296d4e8915ff8875","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/duplicati/duplicati/pulls/1601/commits","review_comments_url":"https://api.github.com/repos/duplicati/duplicati/pulls/1601/comments","review_comment_url":"https://api.github.com/repos/duplicati/duplicati/pulls/comments{/number}","comments_url":"https://api.github.com/repos/duplicati/duplicati/issues/1601/comments","statuses_url":"https://api.github.com/repos/duplicati/duplicati/statuses/e22a47ef839687f8b95bf1cb73e283ba8c2a0084","head":{"label":"FootStark:ParalellFileScanning","ref":"ParalellFileScanning","sha":"e22a47ef839687f8b95bf1cb73e283ba8c2a0084","user":{"login":"FootStark","id":17236227,"avatar_url":"https://avatars.githubusercontent.com/u/17236227?v=3","gravatar_id":"","url":"https://api.github.com/users/FootStark","html_url":"https://github.com/FootStark","followers_url":"https://api.github.com/users/FootStark/followers","following_url":"https://api.github.com/users/FootStark/following{/other_user}","gists_url":"https://api.github.com/users/FootStark/gists{/gist_id}","starred_url":"https://api.github.com/users/FootStark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FootStark/subscriptions","organizations_url":"https://api.github.com/users/FootStark/orgs","repos_url":"https://api.github.com/users/FootStark/repos","events_url":"https://api.github.com/users/FootStark/events{/privacy}","received_events_url":"https://api.github.com/users/FootStark/received_events","type":"User","site_admin":false},"repo":{"id":51706078,"name":"duplicati","full_name":"FootStark/duplicati","owner":{"login":"FootStark","id":17236227,"avatar_url":"https://avatars.githubusercontent.com/u/17236227?v=3","gravatar_id":"","url":"https://api.github.com/users/FootStark","html_url":"https://github.com/FootStark","followers_url":"https://api.github.com/users/FootStark/followers","following_url":"https://api.github.com/users/FootStark/following{/other_user}","gists_url":"https://api.github.com/users/FootStark/gists{/gist_id}","starred_url":"https://api.github.com/users/FootStark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FootStark/subscriptions","organizations_url":"https://api.github.com/users/FootStark/orgs","repos_url":"https://api.github.com/users/FootStark/repos","events_url":"https://api.github.com/users/FootStark/events{/privacy}","received_events_url":"https://api.github.com/users/FootStark/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/FootStark/duplicati","description":"Store securely encrypted backups on cloud storage services!","fork":true,"url":"https://api.github.com/repos/FootStark/duplicati","forks_url":"https://api.github.com/repos/FootStark/duplicati/forks","keys_url":"https://api.github.com/repos/FootStark/duplicati/keys{/key_id}","collaborators_url":"https://api.github.com/repos/FootStark/duplicati/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/FootStark/duplicati/teams","hooks_url":"https://api.github.com/repos/FootStark/duplicati/hooks","issue_events_url":"https://api.github.com/repos/FootStark/duplicati/issues/events{/number}","events_url":"https://api.github.com/repos/FootStark/duplicati/events","assignees_url":"https://api.github.com/repos/FootStark/duplicati/assignees{/user}","branches_url":"https://api.github.com/repos/FootStark/duplicati/branches{/branch}","tags_url":"https://api.github.com/repos/FootStark/duplicati/tags","blobs_url":"https://api.github.com/repos/FootStark/duplicati/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/FootStark/duplicati/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/FootStark/duplicati/git/refs{/sha}","trees_url":"https://api.github.com/repos/FootStark/duplicati/git/trees{/sha}","statuses_url":"https://api.github.com/repos/FootStark/duplicati/statuses/{sha}","languages_url":"https://api.github.com/repos/FootStark/duplicati/languages","stargazers_url":"https://api.github.com/repos/FootStark/duplicati/stargazers","contributors_url":"https://api.github.com/repos/FootStark/duplicati/contributors","subscribers_url":"https://api.github.com/repos/FootStark/duplicati/subscribers","subscription_url":"https://api.github.com/repos/FootStark/duplicati/subscription","commits_url":"https://api.github.com/repos/FootStark/duplicati/commits{/sha}","git_commits_url":"https://api.github.com/repos/FootStark/duplicati/git/commits{/sha}","comments_url":"https://api.github.com/repos/FootStark/duplicati/comments{/number}","issue_comment_url":"https://api.github.com/repos/FootStark/duplicati/issues/comments{/number}","contents_url":"https://api.github.com/repos/FootStark/duplicati/contents/{+path}","compare_url":"https://api.github.com/repos/FootStark/duplicati/compare/{base}...{head}","merges_url":"https://api.github.com/repos/FootStark/duplicati/merges","archive_url":"https://api.github.com/repos/FootStark/duplicati/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/FootStark/duplicati/downloads","issues_url":"https://api.github.com/repos/FootStark/duplicati/issues{/number}","pulls_url":"https://api.github.com/repos/FootStark/duplicati/pulls{/number}","milestones_url":"https://api.github.com/repos/FootStark/duplicati/milestones{/number}","notifications_url":"https://api.github.com/repos/FootStark/duplicati/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/FootStark/duplicati/labels{/name}","releases_url":"https://api.github.com/repos/FootStark/duplicati/releases{/id}","deployments_url":"https://api.github.com/repos/FootStark/duplicati/deployments","created_at":"2016-02-14T18:00:32Z","updated_at":"2016-02-14T18:00:36Z","pushed_at":"2016-03-05T09:57:16Z","git_url":"git://github.com/FootStark/duplicati.git","ssh_url":"git@github.com:FootStark/duplicati.git","clone_url":"https://github.com/FootStark/duplicati.git","svn_url":"https://github.com/FootStark/duplicati","homepage":"","size":60091,"stargazers_count":0,"watchers_count":0,"language":"C#","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"duplicati:master","ref":"master","sha":"5030eeb21f417c59304e6cf4c581ad6d915869ae","user":{"login":"duplicati","id":8270231,"avatar_url":"https://avatars.githubusercontent.com/u/8270231?v=3","gravatar_id":"","url":"https://api.github.com/users/duplicati","html_url":"https://github.com/duplicati","followers_url":"https://api.github.com/users/duplicati/followers","following_url":"https://api.github.com/users/duplicati/following{/other_user}","gists_url":"https://api.github.com/users/duplicati/gists{/gist_id}","starred_url":"https://api.github.com/users/duplicati/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duplicati/subscriptions","organizations_url":"https://api.github.com/users/duplicati/orgs","repos_url":"https://api.github.com/users/duplicati/repos","events_url":"https://api.github.com/users/duplicati/events{/privacy}","received_events_url":"https://api.github.com/users/duplicati/received_events","type":"Organization","site_admin":false},"repo":{"id":22636263,"name":"duplicati","full_name":"duplicati/duplicati","owner":{"login":"duplicati","id":8270231,"avatar_url":"https://avatars.githubusercontent.com/u/8270231?v=3","gravatar_id":"","url":"https://api.github.com/users/duplicati","html_url":"https://github.com/duplicati","followers_url":"https://api.github.com/users/duplicati/followers","following_url":"https://api.github.com/users/duplicati/following{/other_user}","gists_url":"https://api.github.com/users/duplicati/gists{/gist_id}","starred_url":"https://api.github.com/users/duplicati/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duplicati/subscriptions","organizations_url":"https://api.github.com/users/duplicati/orgs","repos_url":"https://api.github.com/users/duplicati/repos","events_url":"https://api.github.com/users/duplicati/events{/privacy}","received_events_url":"https://api.github.com/users/duplicati/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/duplicati/duplicati","description":"Store securely encrypted backups on cloud storage services!","fork":false,"url":"https://api.github.com/repos/duplicati/duplicati","forks_url":"https://api.github.com/repos/duplicati/duplicati/forks","keys_url":"https://api.github.com/repos/duplicati/duplicati/keys{/key_id}","collaborators_url":"https://api.github.com/repos/duplicati/duplicati/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/duplicati/duplicati/teams","hooks_url":"https://api.github.com/repos/duplicati/duplicati/hooks","issue_events_url":"https://api.github.com/repos/duplicati/duplicati/issues/events{/number}","events_url":"https://api.github.com/repos/duplicati/duplicati/events","assignees_url":"https://api.github.com/repos/duplicati/duplicati/assignees{/user}","branches_url":"https://api.github.com/repos/duplicati/duplicati/branches{/branch}","tags_url":"https://api.github.com/repos/duplicati/duplicati/tags","blobs_url":"https://api.github.com/repos/duplicati/duplicati/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/duplicati/duplicati/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/duplicati/duplicati/git/refs{/sha}","trees_url":"https://api.github.com/repos/duplicati/duplicati/git/trees{/sha}","statuses_url":"https://api.github.com/repos/duplicati/duplicati/statuses/{sha}","languages_url":"https://api.github.com/repos/duplicati/duplicati/languages","stargazers_url":"https://api.github.com/repos/duplicati/duplicati/stargazers","contributors_url":"https://api.github.com/repos/duplicati/duplicati/contributors","subscribers_url":"https://api.github.com/repos/duplicati/duplicati/subscribers","subscription_url":"https://api.github.com/repos/duplicati/duplicati/subscription","commits_url":"https://api.github.com/repos/duplicati/duplicati/commits{/sha}","git_commits_url":"https://api.github.com/repos/duplicati/duplicati/git/commits{/sha}","comments_url":"https://api.github.com/repos/duplicati/duplicati/comments{/number}","issue_comment_url":"https://api.github.com/repos/duplicati/duplicati/issues/comments{/number}","contents_url":"https://api.github.com/repos/duplicati/duplicati/contents/{+path}","compare_url":"https://api.github.com/repos/duplicati/duplicati/compare/{base}...{head}","merges_url":"https://api.github.com/repos/duplicati/duplicati/merges","archive_url":"https://api.github.com/repos/duplicati/duplicati/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/duplicati/duplicati/downloads","issues_url":"https://api.github.com/repos/duplicati/duplicati/issues{/number}","pulls_url":"https://api.github.com/repos/duplicati/duplicati/pulls{/number}","milestones_url":"https://api.github.com/repos/duplicati/duplicati/milestones{/number}","notifications_url":"https://api.github.com/repos/duplicati/duplicati/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/duplicati/duplicati/labels{/name}","releases_url":"https://api.github.com/repos/duplicati/duplicati/releases{/id}","deployments_url":"https://api.github.com/repos/duplicati/duplicati/deployments","created_at":"2014-08-05T08:28:16Z","updated_at":"2016-03-06T18:38:55Z","pushed_at":"2016-03-06T23:09:05Z","git_url":"git://github.com/duplicati/duplicati.git","ssh_url":"git@github.com:duplicati/duplicati.git","clone_url":"https://github.com/duplicati/duplicati.git","svn_url":"https://github.com/duplicati/duplicati","homepage":"","size":61688,"stargazers_count":447,"watchers_count":447,"language":"C#","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":96,"mirror_url":null,"open_issues_count":666,"forks":96,"open_issues":666,"watchers":447,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/duplicati/duplicati/pulls/1601"},"html":{"href":"https://github.com/duplicati/duplicati/pull/1601"},"issue":{"href":"https://api.github.com/repos/duplicati/duplicati/issues/1601"},"comments":{"href":"https://api.github.com/repos/duplicati/duplicati/issues/1601/comments"},"review_comments":{"href":"https://api.github.com/repos/duplicati/duplicati/pulls/1601/comments"},"review_comment":{"href":"https://api.github.com/repos/duplicati/duplicati/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/duplicati/duplicati/pulls/1601/commits"},"statuses":{"href":"https://api.github.com/repos/duplicati/duplicati/statuses/e22a47ef839687f8b95bf1cb73e283ba8c2a0084"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"kenkendk","id":1510755,"avatar_url":"https://avatars.githubusercontent.com/u/1510755?v=3","gravatar_id":"","url":"https://api.github.com/users/kenkendk","html_url":"https://github.com/kenkendk","followers_url":"https://api.github.com/users/kenkendk/followers","following_url":"https://api.github.com/users/kenkendk/following{/other_user}","gists_url":"https://api.github.com/users/kenkendk/gists{/gist_id}","starred_url":"https://api.github.com/users/kenkendk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kenkendk/subscriptions","organizations_url":"https://api.github.com/users/kenkendk/orgs","repos_url":"https://api.github.com/users/kenkendk/repos","events_url":"https://api.github.com/users/kenkendk/events{/privacy}","received_events_url":"https://api.github.com/users/kenkendk/received_events","type":"User","site_admin":false},"comments":2,"review_comments":0,"commits":5,"additions":117,"deletions":60,"changed_files":2}}