{"action":"created","comment":{"url":"https://api.github.com/repos/folbricht/desync/pulls/comments/207086278","pull_request_review_id":142626199,"id":207086278,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwNzA4NjI3OA==","diff_hunk":"@@ -152,33 +165,34 @@ func (c *pChunker) start(ctx context.Context) {\n \t\t\treturn\n \t\tdefault: // We weren't asked to stop and weren't interrupted, carry on\n \t\t}\n-\t\tstart, b, err := c.chunker.Next()\n+\t\tchunk, err := c.chunker.Next()\n \t\tif err != nil {\n \t\t\tc.err = err\n \t\t\treturn\n \t\t}\n \t\tc.stats.incProduced()\n-\t\tstart += c.offset\n-\t\tif len(b) == 0 {\n+\t\tchunk.Start += c.offset\n+\t\tif len(chunk.Data) == 0 {\n \t\t\t// TODO: If this worker reached the end of the stream and it's not the\n \t\t\t// last one, we should probable stop all following workers. Meh, shouldn't\n \t\t\t// be happening for large file or save significant CPU for small ones.\n \t\t\tc.eof = true\n \t\t\treturn\n \t\t}\n-\t\t// Calculate the chunk ID\n-\t\tid := sha512.Sum512_256(b)\n-\n \t\t// Store it in our bucket\n-\t\tchunk := IndexChunk{Start: start, Size: uint64(len(b)), ID: id, b: b}\n-\t\tc.results <- chunk\n+\t\tci := NewChunkIndex(chunk)\n+\t\tc.results <- ci\n \n \t\t// Check if the next worker already has this chunk, at which point we stop\n \t\t// here and let the next continue\n-\t\tif c.next != nil && c.next.syncWith(chunk) {\n+\t\tif c.next != nil && c.next.syncWith(ci) {\n \t\t\treturn\n \t\t}\n \n+\t\tif c.s != nil {\n+\t\t\tc.err = c.s.StoreChunk(chunk)","path":"make.go","position":94,"original_position":94,"commit_id":"d4698b86ffb24fbae6f3cdf6a867bdf33a7802f0","original_commit_id":"f47fbba25594ff5607cbb4cdb4c2a105c45bc0be","user":{"login":"folbricht","id":3505107,"node_id":"MDQ6VXNlcjM1MDUxMDc=","avatar_url":"https://avatars2.githubusercontent.com/u/3505107?v=4","gravatar_id":"","url":"https://api.github.com/users/folbricht","html_url":"https://github.com/folbricht","followers_url":"https://api.github.com/users/folbricht/followers","following_url":"https://api.github.com/users/folbricht/following{/other_user}","gists_url":"https://api.github.com/users/folbricht/gists{/gist_id}","starred_url":"https://api.github.com/users/folbricht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/folbricht/subscriptions","organizations_url":"https://api.github.com/users/folbricht/orgs","repos_url":"https://api.github.com/users/folbricht/repos","events_url":"https://api.github.com/users/folbricht/events{/privacy}","received_events_url":"https://api.github.com/users/folbricht/received_events","type":"User","site_admin":false},"body":"The chunks produced during parallel chunking aren't confirmed yet and shouldn't be stored yet.","created_at":"2018-08-02T02:32:00Z","updated_at":"2018-08-02T02:40:17Z","html_url":"https://github.com/folbricht/desync/pull/46#discussion_r207086278","pull_request_url":"https://api.github.com/repos/folbricht/desync/pulls/46","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/folbricht/desync/pulls/comments/207086278"},"html":{"href":"https://github.com/folbricht/desync/pull/46#discussion_r207086278"},"pull_request":{"href":"https://api.github.com/repos/folbricht/desync/pulls/46"}}},"pull_request":{"url":"https://api.github.com/repos/folbricht/desync/pulls/46","id":205008443,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA1MDA4NDQz","html_url":"https://github.com/folbricht/desync/pull/46","diff_url":"https://github.com/folbricht/desync/pull/46.diff","patch_url":"https://github.com/folbricht/desync/pull/46.patch","issue_url":"https://api.github.com/repos/folbricht/desync/issues/46","number":46,"state":"open","locked":false,"title":"Encapsulate chunk storage in a single place","user":{"login":"angelcar","id":41758686,"node_id":"MDQ6VXNlcjQxNzU4Njg2","avatar_url":"https://avatars3.githubusercontent.com/u/41758686?v=4","gravatar_id":"","url":"https://api.github.com/users/angelcar","html_url":"https://github.com/angelcar","followers_url":"https://api.github.com/users/angelcar/followers","following_url":"https://api.github.com/users/angelcar/following{/other_user}","gists_url":"https://api.github.com/users/angelcar/gists{/gist_id}","starred_url":"https://api.github.com/users/angelcar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angelcar/subscriptions","organizations_url":"https://api.github.com/users/angelcar/orgs","repos_url":"https://api.github.com/users/angelcar/repos","events_url":"https://api.github.com/users/angelcar/events{/privacy}","received_events_url":"https://api.github.com/users/angelcar/received_events","type":"User","site_admin":false},"body":"# Changes\r\nThis change is just a refactor to encapsulate chunk storage in a single place. Now `chop`, `make` and `tar` commands use the same piece of code to store chunks. The duplicated code being encapsulated was repeated in a couple of places and that already made a bug fix ineffective since it was applied to one place but not the other (copy paste was needed). For more info about the bug please see https://github.com/folbricht/desync/issues/37\r\n\r\nIn addition, I think chop.go was unnecessarily re-reading the data file [here](https://github.com/folbricht/desync/blob/master/chop.go#L59) since `IndexFromFile` was already chunking and reading the file [here](https://github.com/folbricht/desync/blob/master/make.go#L62). I considered this duplicated code. The only funny thing I had to do to make this work was to declare the `[]byte` directly in `IndexChunk` so that I can pass it around between functions. I realize that this might not be what you intended since `IndexChunk` contains only things that you are writing to the index file, but I figured that as long as `Index.WriteTo` ignores this new field, its all good.\r\n\r\nCan you confirm I am not missing anything here?\r\n\r\nCheers :)\r\n\r\n# Testing done\r\n\r\n\r\n* desync unit tests passed\r\n* Chopped and assembled back a sample file\r\n* Executed tar/untar and make/extract with new files and the tool works as expected\r\n* Tested for backwards compatibility by chopping a file using a binary with my changes in place and assembled back the file using a binary without my changes. This proves that my changes are backwards compatible.","created_at":"2018-07-31T06:30:18Z","updated_at":"2018-08-02T02:40:17Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6b96f10a01db45a486e391c8a756b87355701684","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/folbricht/desync/pulls/46/commits","review_comments_url":"https://api.github.com/repos/folbricht/desync/pulls/46/comments","review_comment_url":"https://api.github.com/repos/folbricht/desync/pulls/comments{/number}","comments_url":"https://api.github.com/repos/folbricht/desync/issues/46/comments","statuses_url":"https://api.github.com/repos/folbricht/desync/statuses/d4698b86ffb24fbae6f3cdf6a867bdf33a7802f0","head":{"label":"angelcar:refactor/chunk-encapsulation","ref":"refactor/chunk-encapsulation","sha":"d4698b86ffb24fbae6f3cdf6a867bdf33a7802f0","user":{"login":"angelcar","id":41758686,"node_id":"MDQ6VXNlcjQxNzU4Njg2","avatar_url":"https://avatars3.githubusercontent.com/u/41758686?v=4","gravatar_id":"","url":"https://api.github.com/users/angelcar","html_url":"https://github.com/angelcar","followers_url":"https://api.github.com/users/angelcar/followers","following_url":"https://api.github.com/users/angelcar/following{/other_user}","gists_url":"https://api.github.com/users/angelcar/gists{/gist_id}","starred_url":"https://api.github.com/users/angelcar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angelcar/subscriptions","organizations_url":"https://api.github.com/users/angelcar/orgs","repos_url":"https://api.github.com/users/angelcar/repos","events_url":"https://api.github.com/users/angelcar/events{/privacy}","received_events_url":"https://api.github.com/users/angelcar/received_events","type":"User","site_admin":false},"repo":{"id":142483805,"node_id":"MDEwOlJlcG9zaXRvcnkxNDI0ODM4MDU=","name":"desync","full_name":"angelcar/desync","owner":{"login":"angelcar","id":41758686,"node_id":"MDQ6VXNlcjQxNzU4Njg2","avatar_url":"https://avatars3.githubusercontent.com/u/41758686?v=4","gravatar_id":"","url":"https://api.github.com/users/angelcar","html_url":"https://github.com/angelcar","followers_url":"https://api.github.com/users/angelcar/followers","following_url":"https://api.github.com/users/angelcar/following{/other_user}","gists_url":"https://api.github.com/users/angelcar/gists{/gist_id}","starred_url":"https://api.github.com/users/angelcar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angelcar/subscriptions","organizations_url":"https://api.github.com/users/angelcar/orgs","repos_url":"https://api.github.com/users/angelcar/repos","events_url":"https://api.github.com/users/angelcar/events{/privacy}","received_events_url":"https://api.github.com/users/angelcar/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/angelcar/desync","description":"Alternative casync implementation","fork":true,"url":"https://api.github.com/repos/angelcar/desync","forks_url":"https://api.github.com/repos/angelcar/desync/forks","keys_url":"https://api.github.com/repos/angelcar/desync/keys{/key_id}","collaborators_url":"https://api.github.com/repos/angelcar/desync/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/angelcar/desync/teams","hooks_url":"https://api.github.com/repos/angelcar/desync/hooks","issue_events_url":"https://api.github.com/repos/angelcar/desync/issues/events{/number}","events_url":"https://api.github.com/repos/angelcar/desync/events","assignees_url":"https://api.github.com/repos/angelcar/desync/assignees{/user}","branches_url":"https://api.github.com/repos/angelcar/desync/branches{/branch}","tags_url":"https://api.github.com/repos/angelcar/desync/tags","blobs_url":"https://api.github.com/repos/angelcar/desync/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/angelcar/desync/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/angelcar/desync/git/refs{/sha}","trees_url":"https://api.github.com/repos/angelcar/desync/git/trees{/sha}","statuses_url":"https://api.github.com/repos/angelcar/desync/statuses/{sha}","languages_url":"https://api.github.com/repos/angelcar/desync/languages","stargazers_url":"https://api.github.com/repos/angelcar/desync/stargazers","contributors_url":"https://api.github.com/repos/angelcar/desync/contributors","subscribers_url":"https://api.github.com/repos/angelcar/desync/subscribers","subscription_url":"https://api.github.com/repos/angelcar/desync/subscription","commits_url":"https://api.github.com/repos/angelcar/desync/commits{/sha}","git_commits_url":"https://api.github.com/repos/angelcar/desync/git/commits{/sha}","comments_url":"https://api.github.com/repos/angelcar/desync/comments{/number}","issue_comment_url":"https://api.github.com/repos/angelcar/desync/issues/comments{/number}","contents_url":"https://api.github.com/repos/angelcar/desync/contents/{+path}","compare_url":"https://api.github.com/repos/angelcar/desync/compare/{base}...{head}","merges_url":"https://api.github.com/repos/angelcar/desync/merges","archive_url":"https://api.github.com/repos/angelcar/desync/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/angelcar/desync/downloads","issues_url":"https://api.github.com/repos/angelcar/desync/issues{/number}","pulls_url":"https://api.github.com/repos/angelcar/desync/pulls{/number}","milestones_url":"https://api.github.com/repos/angelcar/desync/milestones{/number}","notifications_url":"https://api.github.com/repos/angelcar/desync/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/angelcar/desync/labels{/name}","releases_url":"https://api.github.com/repos/angelcar/desync/releases{/id}","deployments_url":"https://api.github.com/repos/angelcar/desync/deployments","created_at":"2018-07-26T19:14:57Z","updated_at":"2018-07-31T06:05:42Z","pushed_at":"2018-08-01T21:28:33Z","git_url":"git://github.com/angelcar/desync.git","ssh_url":"git@github.com:angelcar/desync.git","clone_url":"https://github.com/angelcar/desync.git","svn_url":"https://github.com/angelcar/desync","homepage":"","size":1377,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"bsd-3-clause","name":"BSD 3-Clause \"New\" or \"Revised\" License","spdx_id":"BSD-3-Clause","url":"https://api.github.com/licenses/bsd-3-clause","node_id":"MDc6TGljZW5zZTU="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"folbricht:master","ref":"master","sha":"badf03f3ea7c4bd51f89a16523ce1d07223756dc","user":{"login":"folbricht","id":3505107,"node_id":"MDQ6VXNlcjM1MDUxMDc=","avatar_url":"https://avatars2.githubusercontent.com/u/3505107?v=4","gravatar_id":"","url":"https://api.github.com/users/folbricht","html_url":"https://github.com/folbricht","followers_url":"https://api.github.com/users/folbricht/followers","following_url":"https://api.github.com/users/folbricht/following{/other_user}","gists_url":"https://api.github.com/users/folbricht/gists{/gist_id}","starred_url":"https://api.github.com/users/folbricht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/folbricht/subscriptions","organizations_url":"https://api.github.com/users/folbricht/orgs","repos_url":"https://api.github.com/users/folbricht/repos","events_url":"https://api.github.com/users/folbricht/events{/privacy}","received_events_url":"https://api.github.com/users/folbricht/received_events","type":"User","site_admin":false},"repo":{"id":110053009,"node_id":"MDEwOlJlcG9zaXRvcnkxMTAwNTMwMDk=","name":"desync","full_name":"folbricht/desync","owner":{"login":"folbricht","id":3505107,"node_id":"MDQ6VXNlcjM1MDUxMDc=","avatar_url":"https://avatars2.githubusercontent.com/u/3505107?v=4","gravatar_id":"","url":"https://api.github.com/users/folbricht","html_url":"https://github.com/folbricht","followers_url":"https://api.github.com/users/folbricht/followers","following_url":"https://api.github.com/users/folbricht/following{/other_user}","gists_url":"https://api.github.com/users/folbricht/gists{/gist_id}","starred_url":"https://api.github.com/users/folbricht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/folbricht/subscriptions","organizations_url":"https://api.github.com/users/folbricht/orgs","repos_url":"https://api.github.com/users/folbricht/repos","events_url":"https://api.github.com/users/folbricht/events{/privacy}","received_events_url":"https://api.github.com/users/folbricht/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/folbricht/desync","description":"Alternative casync implementation","fork":false,"url":"https://api.github.com/repos/folbricht/desync","forks_url":"https://api.github.com/repos/folbricht/desync/forks","keys_url":"https://api.github.com/repos/folbricht/desync/keys{/key_id}","collaborators_url":"https://api.github.com/repos/folbricht/desync/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/folbricht/desync/teams","hooks_url":"https://api.github.com/repos/folbricht/desync/hooks","issue_events_url":"https://api.github.com/repos/folbricht/desync/issues/events{/number}","events_url":"https://api.github.com/repos/folbricht/desync/events","assignees_url":"https://api.github.com/repos/folbricht/desync/assignees{/user}","branches_url":"https://api.github.com/repos/folbricht/desync/branches{/branch}","tags_url":"https://api.github.com/repos/folbricht/desync/tags","blobs_url":"https://api.github.com/repos/folbricht/desync/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/folbricht/desync/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/folbricht/desync/git/refs{/sha}","trees_url":"https://api.github.com/repos/folbricht/desync/git/trees{/sha}","statuses_url":"https://api.github.com/repos/folbricht/desync/statuses/{sha}","languages_url":"https://api.github.com/repos/folbricht/desync/languages","stargazers_url":"https://api.github.com/repos/folbricht/desync/stargazers","contributors_url":"https://api.github.com/repos/folbricht/desync/contributors","subscribers_url":"https://api.github.com/repos/folbricht/desync/subscribers","subscription_url":"https://api.github.com/repos/folbricht/desync/subscription","commits_url":"https://api.github.com/repos/folbricht/desync/commits{/sha}","git_commits_url":"https://api.github.com/repos/folbricht/desync/git/commits{/sha}","comments_url":"https://api.github.com/repos/folbricht/desync/comments{/number}","issue_comment_url":"https://api.github.com/repos/folbricht/desync/issues/comments{/number}","contents_url":"https://api.github.com/repos/folbricht/desync/contents/{+path}","compare_url":"https://api.github.com/repos/folbricht/desync/compare/{base}...{head}","merges_url":"https://api.github.com/repos/folbricht/desync/merges","archive_url":"https://api.github.com/repos/folbricht/desync/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/folbricht/desync/downloads","issues_url":"https://api.github.com/repos/folbricht/desync/issues{/number}","pulls_url":"https://api.github.com/repos/folbricht/desync/pulls{/number}","milestones_url":"https://api.github.com/repos/folbricht/desync/milestones{/number}","notifications_url":"https://api.github.com/repos/folbricht/desync/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/folbricht/desync/labels{/name}","releases_url":"https://api.github.com/repos/folbricht/desync/releases{/id}","deployments_url":"https://api.github.com/repos/folbricht/desync/deployments","created_at":"2017-11-09T01:48:17Z","updated_at":"2018-07-31T00:56:19Z","pushed_at":"2018-08-01T21:28:34Z","git_url":"git://github.com/folbricht/desync.git","ssh_url":"git@github.com:folbricht/desync.git","clone_url":"https://github.com/folbricht/desync.git","svn_url":"https://github.com/folbricht/desync","homepage":"","size":1361,"stargazers_count":18,"watchers_count":18,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"bsd-3-clause","name":"BSD 3-Clause \"New\" or \"Revised\" License","spdx_id":"BSD-3-Clause","url":"https://api.github.com/licenses/bsd-3-clause","node_id":"MDc6TGljZW5zZTU="},"forks":8,"open_issues":5,"watchers":18,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/folbricht/desync/pulls/46"},"html":{"href":"https://github.com/folbricht/desync/pull/46"},"issue":{"href":"https://api.github.com/repos/folbricht/desync/issues/46"},"comments":{"href":"https://api.github.com/repos/folbricht/desync/issues/46/comments"},"review_comments":{"href":"https://api.github.com/repos/folbricht/desync/pulls/46/comments"},"review_comment":{"href":"https://api.github.com/repos/folbricht/desync/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/folbricht/desync/pulls/46/commits"},"statuses":{"href":"https://api.github.com/repos/folbricht/desync/statuses/d4698b86ffb24fbae6f3cdf6a867bdf33a7802f0"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"folbricht"}}