{"action":"created","comment":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/comments/61119669","id":61119669,"diff_hunk":"@@ -591,6 +602,28 @@ def get_fd(self, segment):\n             self.fds[segment] = fd\n             return fd\n \n+    def _sync_and_close(self, fd):\n+        fd.flush()\n+        os.fdatasync(fd.fileno())","path":"borg/repository.py","position":50,"original_position":50,"commit_id":"f987da84052c75bba67596d3831ace6f1a2e1f56","original_commit_id":"91e8bb4b91ed7862f734706d1bcb1b2a2298b21a","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"body":"uhoh, while searching this i found that OSX doesn't make the guarantee that fsync really waits until all data arrived on disk.","created_at":"2016-04-26T16:37:33Z","updated_at":"2016-04-26T16:37:33Z","html_url":"https://github.com/borgbackup/borg/pull/985#discussion_r61119669","pull_request_url":"https://api.github.com/repos/borgbackup/borg/pulls/985","_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments/61119669"},"html":{"href":"https://github.com/borgbackup/borg/pull/985#discussion_r61119669"},"pull_request":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/985"}}},"pull_request":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/985","id":67886167,"html_url":"https://github.com/borgbackup/borg/pull/985","diff_url":"https://github.com/borgbackup/borg/pull/985.diff","patch_url":"https://github.com/borgbackup/borg/pull/985.patch","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/985","number":985,"state":"open","locked":false,"title":"wip, Improved LoggedIO write performance","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"body":"Also see https://github.com/borgbackup/borg/issues/45#issuecomment-213715080\r\n\r\nSummary\r\n\r\n\r\nTest set:\r\n\r\n*  images + XMP metadata (for some folders)\r\n* average file size ~3.5 MB\r\n* 5388 files\r\n* 19.49 GB\r\n\r\n```\r\n                        Throughput     %CPU      total CPU time\r\n1.0.2 / master ........ 24 MB/s ...... 33 % .... 280 s\r\nfastlio none .......... 52 MB/s ...... 62 % .... 237 s\r\nfastlio none64 ........ 54 MB/s ...... 55 % .... 205 s\r\n```\r\n\r\nAll are I/O-bound, but fastlio is less so. free(1) shows that master branch doesn't\r\nlet the kernel use it's buffers (due to fsync after each segment) efficiently, while\r\nfastlio does. Profiling shows that for fastlio fsync() makes no significant wall-clock\r\nappereance, while it is a large factor for master.\r\n\r\nThis patch doesn't (or shouldn't) trash the page cache, as it immediately tells the kernel after writing\r\nthat the data is not needed anymore. The fsync() comes much later, at a point where it seems\r\nthe kernel already wrote everything to disk on it's own accord. Hence these fsync()s return\r\nimmediately, instead of blocking.\r\n\r\nWhile in my tests none and none64 did not make a tangible difference, none64 would be useful\r\nwhere fast IO is available but the CPU isn't very fast (=NAS), since the average CPU loads\r\nshow it uses much less CPU time.\r\n\r\nFor machines running encrypted repositories with hardware AES support (AES-NI for example)\r\nthe throughput should scale similar. For machines with no hardware AES support the difference\r\nwill be less pronounced.\r\n","created_at":"2016-04-26T13:32:51Z","updated_at":"2016-04-26T16:37:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b56373e281eef44ea8f39f3c4e7a0eacb09ed31a","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/borgbackup/borg/pulls/985/commits","review_comments_url":"https://api.github.com/repos/borgbackup/borg/pulls/985/comments","review_comment_url":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/985/comments","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/f987da84052c75bba67596d3831ace6f1a2e1f56","head":{"label":"enkore:feature/fastlio","ref":"feature/fastlio","sha":"f987da84052c75bba67596d3831ace6f1a2e1f56","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"repo":{"id":53583744,"name":"borg","full_name":"enkore/borg","owner":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/enkore/borg","description":"Dev fork","fork":true,"url":"https://api.github.com/repos/enkore/borg","forks_url":"https://api.github.com/repos/enkore/borg/forks","keys_url":"https://api.github.com/repos/enkore/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/enkore/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/enkore/borg/teams","hooks_url":"https://api.github.com/repos/enkore/borg/hooks","issue_events_url":"https://api.github.com/repos/enkore/borg/issues/events{/number}","events_url":"https://api.github.com/repos/enkore/borg/events","assignees_url":"https://api.github.com/repos/enkore/borg/assignees{/user}","branches_url":"https://api.github.com/repos/enkore/borg/branches{/branch}","tags_url":"https://api.github.com/repos/enkore/borg/tags","blobs_url":"https://api.github.com/repos/enkore/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/enkore/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/enkore/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/enkore/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/enkore/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/enkore/borg/languages","stargazers_url":"https://api.github.com/repos/enkore/borg/stargazers","contributors_url":"https://api.github.com/repos/enkore/borg/contributors","subscribers_url":"https://api.github.com/repos/enkore/borg/subscribers","subscription_url":"https://api.github.com/repos/enkore/borg/subscription","commits_url":"https://api.github.com/repos/enkore/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/enkore/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/enkore/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/enkore/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/enkore/borg/contents/{+path}","compare_url":"https://api.github.com/repos/enkore/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/enkore/borg/merges","archive_url":"https://api.github.com/repos/enkore/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/enkore/borg/downloads","issues_url":"https://api.github.com/repos/enkore/borg/issues{/number}","pulls_url":"https://api.github.com/repos/enkore/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/enkore/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/enkore/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/enkore/borg/labels{/name}","releases_url":"https://api.github.com/repos/enkore/borg/releases{/id}","deployments_url":"https://api.github.com/repos/enkore/borg/deployments","created_at":"2016-03-10T12:45:08Z","updated_at":"2016-04-06T21:28:16Z","pushed_at":"2016-04-26T13:31:50Z","git_url":"git://github.com/enkore/borg.git","ssh_url":"git@github.com:enkore/borg.git","clone_url":"https://github.com/enkore/borg.git","svn_url":"https://github.com/enkore/borg","homepage":"","size":3759,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"borgbackup:master","ref":"master","sha":"ddc7687d9e5e5ab706cd27280a0d021f8b39bf89","user":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"repo":{"id":35517126,"name":"borg","full_name":"borgbackup/borg","owner":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/borgbackup/borg","description":"Deduplicating backup program with compression and authenticated encryption.","fork":false,"url":"https://api.github.com/repos/borgbackup/borg","forks_url":"https://api.github.com/repos/borgbackup/borg/forks","keys_url":"https://api.github.com/repos/borgbackup/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/borgbackup/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/borgbackup/borg/teams","hooks_url":"https://api.github.com/repos/borgbackup/borg/hooks","issue_events_url":"https://api.github.com/repos/borgbackup/borg/issues/events{/number}","events_url":"https://api.github.com/repos/borgbackup/borg/events","assignees_url":"https://api.github.com/repos/borgbackup/borg/assignees{/user}","branches_url":"https://api.github.com/repos/borgbackup/borg/branches{/branch}","tags_url":"https://api.github.com/repos/borgbackup/borg/tags","blobs_url":"https://api.github.com/repos/borgbackup/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/borgbackup/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/borgbackup/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/borgbackup/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/borgbackup/borg/languages","stargazers_url":"https://api.github.com/repos/borgbackup/borg/stargazers","contributors_url":"https://api.github.com/repos/borgbackup/borg/contributors","subscribers_url":"https://api.github.com/repos/borgbackup/borg/subscribers","subscription_url":"https://api.github.com/repos/borgbackup/borg/subscription","commits_url":"https://api.github.com/repos/borgbackup/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/borgbackup/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/borgbackup/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/borgbackup/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/borgbackup/borg/contents/{+path}","compare_url":"https://api.github.com/repos/borgbackup/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/borgbackup/borg/merges","archive_url":"https://api.github.com/repos/borgbackup/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/borgbackup/borg/downloads","issues_url":"https://api.github.com/repos/borgbackup/borg/issues{/number}","pulls_url":"https://api.github.com/repos/borgbackup/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/borgbackup/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/borgbackup/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/borgbackup/borg/labels{/name}","releases_url":"https://api.github.com/repos/borgbackup/borg/releases{/id}","deployments_url":"https://api.github.com/repos/borgbackup/borg/deployments","created_at":"2015-05-12T23:10:47Z","updated_at":"2016-04-25T15:31:48Z","pushed_at":"2016-04-26T15:51:31Z","git_url":"git://github.com/borgbackup/borg.git","ssh_url":"git@github.com:borgbackup/borg.git","clone_url":"https://github.com/borgbackup/borg.git","svn_url":"https://github.com/borgbackup/borg","homepage":"https://borgbackup.github.io/","size":4631,"stargazers_count":1096,"watchers_count":1096,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":92,"mirror_url":null,"open_issues_count":150,"forks":92,"open_issues":150,"watchers":1096,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/985"},"html":{"href":"https://github.com/borgbackup/borg/pull/985"},"issue":{"href":"https://api.github.com/repos/borgbackup/borg/issues/985"},"comments":{"href":"https://api.github.com/repos/borgbackup/borg/issues/985/comments"},"review_comments":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/985/comments"},"review_comment":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/985/commits"},"statuses":{"href":"https://api.github.com/repos/borgbackup/borg/statuses/f987da84052c75bba67596d3831ace6f1a2e1f56"}}}}