{"action":"created","comment":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/comments/59296002","id":59296002,"diff_hunk":"@@ -191,11 +188,23 @@ cdef class ChunkIndex(IndexBase):\n         return iter\n \n     def summarize(self):\n-        cdef long long total_size, total_csize, unique_size, unique_csize, total_unique_chunks, total_chunks\n-        hashindex_summarize(self.index, &total_size, &total_csize,\n-                            &unique_size, &unique_csize,\n-                            &total_unique_chunks, &total_chunks)\n-        return total_size, total_csize, unique_size, unique_csize, total_unique_chunks, total_chunks\n+        cdef long long size = 0, csize = 0, unique_size = 0, unique_csize = 0, chunks = 0, unique_chunks = 0\n+        cdef int *values\n+        cdef void *key = NULL\n+\n+        while True:\n+            key = hashindex_next_key(self.index, key)\n+            if not key:\n+                break\n+            unique_chunks += 1\n+            values = <int*> (key + self.key_size)\n+            chunks += _le32toh(values[0])\n+            unique_size += _le32toh(values[1])\n+            unique_csize += _le32toh(values[2])\n+            size += <long long> _le32toh(values[1]) * _le32toh(values[0])\n+            csize += <long long> _le32toh(values[2]) *  _le32toh(values[0])","path":"borg/hashindex.pyx","position":120,"original_position":33,"commit_id":"98b9ca3114bf0a2ae151508ab1073c27db2c1b84","original_commit_id":"d733d37451f14377c08f90f8c6c2e780374b0262","user":{"login":"ThomasWaldmann","id":356103,"avatar_url":"https://avatars.githubusercontent.com/u/356103?v=3","gravatar_id":"","url":"https://api.github.com/users/ThomasWaldmann","html_url":"https://github.com/ThomasWaldmann","followers_url":"https://api.github.com/users/ThomasWaldmann/followers","following_url":"https://api.github.com/users/ThomasWaldmann/following{/other_user}","gists_url":"https://api.github.com/users/ThomasWaldmann/gists{/gist_id}","starred_url":"https://api.github.com/users/ThomasWaldmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ThomasWaldmann/subscriptions","organizations_url":"https://api.github.com/users/ThomasWaldmann/orgs","repos_url":"https://api.github.com/users/ThomasWaldmann/repos","events_url":"https://api.github.com/users/ThomasWaldmann/events{/privacy}","received_events_url":"https://api.github.com/users/ThomasWaldmann/received_events","type":"User","site_admin":false},"body":"long long - can that vary in unwanted ways also?","created_at":"2016-04-11T22:57:08Z","updated_at":"2016-04-11T22:57:08Z","html_url":"https://github.com/borgbackup/borg/pull/888#discussion_r59296002","pull_request_url":"https://api.github.com/repos/borgbackup/borg/pulls/888","_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments/59296002"},"html":{"href":"https://github.com/borgbackup/borg/pull/888#discussion_r59296002"},"pull_request":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/888"}}},"pull_request":{"url":"https://api.github.com/repos/borgbackup/borg/pulls/888","id":66068545,"html_url":"https://github.com/borgbackup/borg/pull/888","diff_url":"https://github.com/borgbackup/borg/pull/888.diff","patch_url":"https://github.com/borgbackup/borg/pull/888.patch","issue_url":"https://api.github.com/repos/borgbackup/borg/issues/888","number":888,"state":"open","locked":false,"title":"Port hashindex_summarize into ChunkIndex.summarize","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"body":"Also adds a test that does\r\n\r\n    checking whether hashindex summarize is sane... probably\r\n\r\nGenerated C code for convenience: https://gist.github.com/enkore/4bdf209d5a16a5deba29d4f101dbf8fd\r\n\r\n-- the loop is essentially the same. Brief performance testing [shows](https://gist.github.com/enkore/4bdf209d5a16a5deba29d4f101dbf8fd#file-perfcheck) no difference in time/space usage, as one would expect.\r\n\r\n#887 is merged into this for master.\r\n\r\n---\r\n\r\n*Also* changes int -> int32_t in those places where it's a format dependency. Cython seems to use what's \"int\" to the underlying C compiler, which is not guaranteed to be 4 bytes (C: >= 2 byte).","created_at":"2016-04-11T21:50:17Z","updated_at":"2016-04-11T22:57:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":"78eecd42b0a2dd16f740219e46044f85209a385b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/borgbackup/borg/pulls/888/commits","review_comments_url":"https://api.github.com/repos/borgbackup/borg/pulls/888/comments","review_comment_url":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}","comments_url":"https://api.github.com/repos/borgbackup/borg/issues/888/comments","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/98b9ca3114bf0a2ae151508ab1073c27db2c1b84","head":{"label":"enkore:feature/cython-summarize","ref":"feature/cython-summarize","sha":"98b9ca3114bf0a2ae151508ab1073c27db2c1b84","user":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"repo":{"id":53583744,"name":"borg","full_name":"enkore/borg","owner":{"login":"enkore","id":1330164,"avatar_url":"https://avatars.githubusercontent.com/u/1330164?v=3","gravatar_id":"","url":"https://api.github.com/users/enkore","html_url":"https://github.com/enkore","followers_url":"https://api.github.com/users/enkore/followers","following_url":"https://api.github.com/users/enkore/following{/other_user}","gists_url":"https://api.github.com/users/enkore/gists{/gist_id}","starred_url":"https://api.github.com/users/enkore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkore/subscriptions","organizations_url":"https://api.github.com/users/enkore/orgs","repos_url":"https://api.github.com/users/enkore/repos","events_url":"https://api.github.com/users/enkore/events{/privacy}","received_events_url":"https://api.github.com/users/enkore/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/enkore/borg","description":"Dev fork","fork":true,"url":"https://api.github.com/repos/enkore/borg","forks_url":"https://api.github.com/repos/enkore/borg/forks","keys_url":"https://api.github.com/repos/enkore/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/enkore/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/enkore/borg/teams","hooks_url":"https://api.github.com/repos/enkore/borg/hooks","issue_events_url":"https://api.github.com/repos/enkore/borg/issues/events{/number}","events_url":"https://api.github.com/repos/enkore/borg/events","assignees_url":"https://api.github.com/repos/enkore/borg/assignees{/user}","branches_url":"https://api.github.com/repos/enkore/borg/branches{/branch}","tags_url":"https://api.github.com/repos/enkore/borg/tags","blobs_url":"https://api.github.com/repos/enkore/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/enkore/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/enkore/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/enkore/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/enkore/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/enkore/borg/languages","stargazers_url":"https://api.github.com/repos/enkore/borg/stargazers","contributors_url":"https://api.github.com/repos/enkore/borg/contributors","subscribers_url":"https://api.github.com/repos/enkore/borg/subscribers","subscription_url":"https://api.github.com/repos/enkore/borg/subscription","commits_url":"https://api.github.com/repos/enkore/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/enkore/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/enkore/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/enkore/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/enkore/borg/contents/{+path}","compare_url":"https://api.github.com/repos/enkore/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/enkore/borg/merges","archive_url":"https://api.github.com/repos/enkore/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/enkore/borg/downloads","issues_url":"https://api.github.com/repos/enkore/borg/issues{/number}","pulls_url":"https://api.github.com/repos/enkore/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/enkore/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/enkore/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/enkore/borg/labels{/name}","releases_url":"https://api.github.com/repos/enkore/borg/releases{/id}","deployments_url":"https://api.github.com/repos/enkore/borg/deployments","created_at":"2016-03-10T12:45:08Z","updated_at":"2016-04-06T21:28:16Z","pushed_at":"2016-04-11T22:13:43Z","git_url":"git://github.com/enkore/borg.git","ssh_url":"git@github.com:enkore/borg.git","clone_url":"https://github.com/enkore/borg.git","svn_url":"https://github.com/enkore/borg","homepage":"","size":3486,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"borgbackup:master","ref":"master","sha":"f8be0cf14c8f76b87758c39a16779c3ab08da79e","user":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"repo":{"id":35517126,"name":"borg","full_name":"borgbackup/borg","owner":{"login":"borgbackup","id":12418060,"avatar_url":"https://avatars.githubusercontent.com/u/12418060?v=3","gravatar_id":"","url":"https://api.github.com/users/borgbackup","html_url":"https://github.com/borgbackup","followers_url":"https://api.github.com/users/borgbackup/followers","following_url":"https://api.github.com/users/borgbackup/following{/other_user}","gists_url":"https://api.github.com/users/borgbackup/gists{/gist_id}","starred_url":"https://api.github.com/users/borgbackup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borgbackup/subscriptions","organizations_url":"https://api.github.com/users/borgbackup/orgs","repos_url":"https://api.github.com/users/borgbackup/repos","events_url":"https://api.github.com/users/borgbackup/events{/privacy}","received_events_url":"https://api.github.com/users/borgbackup/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/borgbackup/borg","description":"Deduplicating backup program with compression and authenticated encryption.","fork":false,"url":"https://api.github.com/repos/borgbackup/borg","forks_url":"https://api.github.com/repos/borgbackup/borg/forks","keys_url":"https://api.github.com/repos/borgbackup/borg/keys{/key_id}","collaborators_url":"https://api.github.com/repos/borgbackup/borg/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/borgbackup/borg/teams","hooks_url":"https://api.github.com/repos/borgbackup/borg/hooks","issue_events_url":"https://api.github.com/repos/borgbackup/borg/issues/events{/number}","events_url":"https://api.github.com/repos/borgbackup/borg/events","assignees_url":"https://api.github.com/repos/borgbackup/borg/assignees{/user}","branches_url":"https://api.github.com/repos/borgbackup/borg/branches{/branch}","tags_url":"https://api.github.com/repos/borgbackup/borg/tags","blobs_url":"https://api.github.com/repos/borgbackup/borg/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/borgbackup/borg/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/borgbackup/borg/git/refs{/sha}","trees_url":"https://api.github.com/repos/borgbackup/borg/git/trees{/sha}","statuses_url":"https://api.github.com/repos/borgbackup/borg/statuses/{sha}","languages_url":"https://api.github.com/repos/borgbackup/borg/languages","stargazers_url":"https://api.github.com/repos/borgbackup/borg/stargazers","contributors_url":"https://api.github.com/repos/borgbackup/borg/contributors","subscribers_url":"https://api.github.com/repos/borgbackup/borg/subscribers","subscription_url":"https://api.github.com/repos/borgbackup/borg/subscription","commits_url":"https://api.github.com/repos/borgbackup/borg/commits{/sha}","git_commits_url":"https://api.github.com/repos/borgbackup/borg/git/commits{/sha}","comments_url":"https://api.github.com/repos/borgbackup/borg/comments{/number}","issue_comment_url":"https://api.github.com/repos/borgbackup/borg/issues/comments{/number}","contents_url":"https://api.github.com/repos/borgbackup/borg/contents/{+path}","compare_url":"https://api.github.com/repos/borgbackup/borg/compare/{base}...{head}","merges_url":"https://api.github.com/repos/borgbackup/borg/merges","archive_url":"https://api.github.com/repos/borgbackup/borg/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/borgbackup/borg/downloads","issues_url":"https://api.github.com/repos/borgbackup/borg/issues{/number}","pulls_url":"https://api.github.com/repos/borgbackup/borg/pulls{/number}","milestones_url":"https://api.github.com/repos/borgbackup/borg/milestones{/number}","notifications_url":"https://api.github.com/repos/borgbackup/borg/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/borgbackup/borg/labels{/name}","releases_url":"https://api.github.com/repos/borgbackup/borg/releases{/id}","deployments_url":"https://api.github.com/repos/borgbackup/borg/deployments","created_at":"2015-05-12T23:10:47Z","updated_at":"2016-04-11T22:41:10Z","pushed_at":"2016-04-11T22:13:44Z","git_url":"git://github.com/borgbackup/borg.git","ssh_url":"git@github.com:borgbackup/borg.git","clone_url":"https://github.com/borgbackup/borg.git","svn_url":"https://github.com/borgbackup/borg","homepage":"https://borgbackup.github.io/","size":3647,"stargazers_count":1058,"watchers_count":1058,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":84,"mirror_url":null,"open_issues_count":118,"forks":84,"open_issues":118,"watchers":1058,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/888"},"html":{"href":"https://github.com/borgbackup/borg/pull/888"},"issue":{"href":"https://api.github.com/repos/borgbackup/borg/issues/888"},"comments":{"href":"https://api.github.com/repos/borgbackup/borg/issues/888/comments"},"review_comments":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/888/comments"},"review_comment":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/borgbackup/borg/pulls/888/commits"},"statuses":{"href":"https://api.github.com/repos/borgbackup/borg/statuses/98b9ca3114bf0a2ae151508ab1073c27db2c1b84"}}}}