{"action":"created","comment":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/comments/63637563","id":63637563,"diff_hunk":"@@ -550,18 +550,37 @@ def cleanup(self, transaction_id):\n             else:\n                 break\n \n-    def is_committed_segment(self, filename):\n+    def is_committed_segment(self, segment):\n         \"\"\"Check if segment ends with a COMMIT_TAG tag\n         \"\"\"\n-        with open(filename, 'rb') as fd:\n+        try:\n+            iterator = self.iter_objects(segment)\n+        except IntegrityError:\n+            return False\n+        with open(self.segment_filename(segment), 'rb') as fd:\n             try:\n                 fd.seek(-self.header_fmt.size, os.SEEK_END)\n             except OSError as e:\n                 # return False if segment file is empty or too small\n                 if e.errno == errno.EINVAL:\n                     return False\n                 raise e\n-            return fd.read(self.header_fmt.size) == self.COMMIT\n+            if fd.read(self.header_fmt.size) != self.COMMIT:\n+                return False\n+        seen_commit = False\n+        while True:\n+            try:\n+                tag, key, offset = next(iterator)\n+            except IntegrityError:\n+                return False\n+            except StopIteration:\n+                break\n+            if tag == TAG_COMMIT:\n+                seen_commit = True\n+                continue\n+            if seen_commit:\n+                raise IntegrityError('Commit not located at end of segment [segment {}, offset {}]'.format(segment, offset))","path":"borg/repository.py","position":45,"original_position":45,"commit_id":"a4ac22bf5598e8eda71d9b35ba0ba1173580c661","original_commit_id":"a4ac22bf5598e8eda71d9b35ba0ba1173580c661","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"body":"the old code would just return False in this case, not raise.","created_at":"2016-05-18T02:35:38Z","updated_at":"2016-05-18T02:35:38Z","html_url":"https://github.com/borgbackup/borg/pull/1052#discussion_r63637563","pull_request_url":"https://api.github.com/repos/borgbackup/borg/pulls/1052","_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments/63637563"},"html":{"href":"https://github.com/borgbackup/borg/pull/1052#discussion_r63637563"},"pull_request":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/1052"}}},"pull_request":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/1052","id":70453596,"html_url":"https://github.com/borgbackup/borg/pull/1052","diff_url":"https://github.com/borgbackup/borg/pull/1052.diff","patch_url":"https://github.com/borgbackup/borg/pull/1052.patch","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/1052","number":1052,"state":"open","locked":false,"title":"Repository: fix commit tags being seen in data","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"body":"I have a slightly different fix for master that doesn't destroy performance with larger segments (i.e. put commits in their own segment)\r\n\r\nProblem description: is_committed_segment() seeked to the end of a segment file and checked whether there is a COMMIT_TAG there. This can also be the case when a PUT ends with a structure that is a COMMIT_TAG and is thus no valid way to check for the commit state of a segment.\r\n\r\nThis is a luck-based bug. You'll need to back something up where a chunk ends with a commit tag (another borg repo) and then that has to end up at the end of a segment file and *then* you'll need a kill/crash and *then* you *can theoretically* get an inconsistent repo (with some other conditions applying as well).\r\n\r\nNow:\r\n\r\n- Early segment magic check by iterator setup in iter_objects\r\n- Sanity check: Do we have a COMMIT_TAG at the end? (<-- this is also the fast-bail-out for practically all uncomitted segments)\r\n- If no: can't be committed\r\n- If yes: iterate objects, check if that's an actual commit tag","created_at":"2016-05-18T00:41:04Z","updated_at":"2016-05-18T02:35:38Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0e1967ba15692ebde08c6be9d1073694e364b836","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/borgbackup/borg/pulls/1052/commits","review_comments_url":"https://api.github.com/repos/borgbackup/borg/pulls/1052/comments","review_comment_url":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/1052/comments","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/a4ac22bf5598e8eda71d9b35ba0ba1173580c661","head":{"label":"enkore:fix/commitcheck","ref":"fix/commitcheck","sha":"a4ac22bf5598e8eda71d9b35ba0ba1173580c661","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"repo":{"id":53583744,"name":"borg","full_name":"enkore/borg","owner":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/enkore/borg","description":"Dev fork","fork":true,"url":"https://api.github.com/repos/enkore/borg","forks_url":"https://api.github.com/repos/enkore/borg/forks","keys_url":"https://api.github.com/repos/enkore/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/enkore/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/enkore/borg/teams","hooks_url":"https://api.github.com/repos/enkore/borg/hooks","issue_events_url":"https://api.github.com/repos/enkore/borg/issues/events{/number}","events_url":"https://api.github.com/repos/enkore/borg/events","assignees_url":"https://api.github.com/repos/enkore/borg/assignees{/user}","branches_url":"https://api.github.com/repos/enkore/borg/branches{/branch}","tags_url":"https://api.github.com/repos/enkore/borg/tags","blobs_url":"https://api.github.com/repos/enkore/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/enkore/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/enkore/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/enkore/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/enkore/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/enkore/borg/languages","stargazers_url":"https://api.github.com/repos/enkore/borg/stargazers","contributors_url":"https://api.github.com/repos/enkore/borg/contributors","subscribers_url":"https://api.github.com/repos/enkore/borg/subscribers","subscription_url":"https://api.github.com/repos/enkore/borg/subscription","commits_url":"https://api.github.com/repos/enkore/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/enkore/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/enkore/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/enkore/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/enkore/borg/contents/{+path}","compare_url":"https://api.github.com/repos/enkore/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/enkore/borg/merges","archive_url":"https://api.github.com/repos/enkore/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/enkore/borg/downloads","issues_url":"https://api.github.com/repos/enkore/borg/issues{/number}","pulls_url":"https://api.github.com/repos/enkore/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/enkore/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/enkore/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/enkore/borg/labels{/name}","releases_url":"https://api.github.com/repos/enkore/borg/releases{/id}","deployments_url":"https://api.github.com/repos/enkore/borg/deployments","created_at":"2016-03-10T12:45:08Z","updated_at":"2016-04-06T21:28:16Z","pushed_at":"2016-05-18T00:56:06Z","git_url":"git://github.com/enkore/borg.git","ssh_url":"git@github.com:enkore/borg.git","clone_url":"https://github.com/enkore/borg.git","svn_url":"https://github.com/enkore/borg","homepage":"","size":3864,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"borgbackup:1.0-maint","ref":"1.0-maint","sha":"eaf0f62022640f64c2e0022e10b3da873ec62f3a","user":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"repo":{"id":35517126,"name":"borg","full_name":"borgbackup/borg","owner":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/borgbackup/borg","description":"Deduplicating backup program with compression and authenticated encryption.","fork":false,"url":"https://api.github.com/repos/borgbackup/borg","forks_url":"https://api.github.com/repos/borgbackup/borg/forks","keys_url":"https://api.github.com/repos/borgbackup/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/borgbackup/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/borgbackup/borg/teams","hooks_url":"https://api.github.com/repos/borgbackup/borg/hooks","issue_events_url":"https://api.github.com/repos/borgbackup/borg/issues/events{/number}","events_url":"https://api.github.com/repos/borgbackup/borg/events","assignees_url":"https://api.github.com/repos/borgbackup/borg/assignees{/user}","branches_url":"https://api.github.com/repos/borgbackup/borg/branches{/branch}","tags_url":"https://api.github.com/repos/borgbackup/borg/tags","blobs_url":"https://api.github.com/repos/borgbackup/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/borgbackup/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/borgbackup/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/borgbackup/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/borgbackup/borg/languages","stargazers_url":"https://api.github.com/repos/borgbackup/borg/stargazers","contributors_url":"https://api.github.com/repos/borgbackup/borg/contributors","subscribers_url":"https://api.github.com/repos/borgbackup/borg/subscribers","subscription_url":"https://api.github.com/repos/borgbackup/borg/subscription","commits_url":"https://api.github.com/repos/borgbackup/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/borgbackup/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/borgbackup/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/borgbackup/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/borgbackup/borg/contents/{+path}","compare_url":"https://api.github.com/repos/borgbackup/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/borgbackup/borg/merges","archive_url":"https://api.github.com/repos/borgbackup/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/borgbackup/borg/downloads","issues_url":"https://api.github.com/repos/borgbackup/borg/issues{/number}","pulls_url":"https://api.github.com/repos/borgbackup/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/borgbackup/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/borgbackup/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/borgbackup/borg/labels{/name}","releases_url":"https://api.github.com/repos/borgbackup/borg/releases{/id}","deployments_url":"https://api.github.com/repos/borgbackup/borg/deployments","created_at":"2015-05-12T23:10:47Z","updated_at":"2016-05-17T16:35:17Z","pushed_at":"2016-05-18T02:28:01Z","git_url":"git://github.com/borgbackup/borg.git","ssh_url":"git@github.com:borgbackup/borg.git","clone_url":"https://github.com/borgbackup/borg.git","svn_url":"https://github.com/borgbackup/borg","homepage":"https://borgbackup.readthedocs.io/","size":4734,"stargazers_count":1157,"watchers_count":1157,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":94,"mirror_url":null,"open_issues_count":161,"forks":94,"open_issues":161,"watchers":1157,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/1052"},"html":{"href":"https://github.com/borgbackup/borg/pull/1052"},"issue":{"href":"https://api.github.com/repos/borgbackup/borg/issues/1052"},"comments":{"href":"https://api.github.com/repos/borgbackup/borg/issues/1052/comments"},"review_comments":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/1052/comments"},"review_comment":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/1052/commits"},"statuses":{"href":"https://api.github.com/repos/borgbackup/borg/statuses/a4ac22bf5598e8eda71d9b35ba0ba1173580c661"}}}}