{"action":"created","comment":{"url":"https://api.github.com/repos/ethersphere/swarm/pulls/comments/293227332","pull_request_review_id":249154237,"id":293227332,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI5MzIyNzMzMg==","diff_hunk":"@@ -19,27 +19,30 @@ package storage\n import (\n \t\"context\"\n \t\"fmt\"\n-\t\"sync/atomic\"\n+\t\"sync\"\n \n \t\"github.com/ethersphere/swarm/chunk\"\n \t\"github.com/ethersphere/swarm/storage/encryption\"\n \t\"golang.org/x/crypto/sha3\"\n )\n \n+const (\n+\tnoOfStorageWorkers = 150\n+)\n+\n type hasherStore struct {\n \tstore     ChunkStore\n \ttag       *chunk.Tag\n \ttoEncrypt bool\n+\tdoWait    sync.Once","path":"storage/hasherstore.go","position":24,"original_position":20,"commit_id":"4c2cfe7bc3ece84c864291b2447c24c3e6c6f074","original_commit_id":"f69133bdfa3fe4f948d7bd9ce7e7ba01166733e3","user":{"login":"nolash","id":9886925,"node_id":"MDQ6VXNlcjk4ODY5MjU=","avatar_url":"https://avatars1.githubusercontent.com/u/9886925?v=4","gravatar_id":"","url":"https://api.github.com/users/nolash","html_url":"https://github.com/nolash","followers_url":"https://api.github.com/users/nolash/followers","following_url":"https://api.github.com/users/nolash/following{/other_user}","gists_url":"https://api.github.com/users/nolash/gists{/gist_id}","starred_url":"https://api.github.com/users/nolash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nolash/subscriptions","organizations_url":"https://api.github.com/users/nolash/orgs","repos_url":"https://api.github.com/users/nolash/repos","events_url":"https://api.github.com/users/nolash/events{/privacy}","received_events_url":"https://api.github.com/users/nolash/received_events","type":"User","site_admin":false},"body":"> I regressed when moving the code through several swarm branches.\r\n\r\n?","created_at":"2019-06-13T06:47:57Z","updated_at":"2019-06-13T06:47:58Z","html_url":"https://github.com/ethersphere/swarm/pull/1395#discussion_r293227332","pull_request_url":"https://api.github.com/repos/ethersphere/swarm/pulls/1395","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/comments/293227332"},"html":{"href":"https://github.com/ethersphere/swarm/pull/1395#discussion_r293227332"},"pull_request":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/1395"}},"in_reply_to_id":291467691},"pull_request":{"url":"https://api.github.com/repos/ethersphere/swarm/pulls/1395","id":278721626,"node_id":"MDExOlB1bGxSZXF1ZXN0Mjc4NzIxNjI2","html_url":"https://github.com/ethersphere/swarm/pull/1395","diff_url":"https://github.com/ethersphere/swarm/pull/1395.diff","patch_url":"https://github.com/ethersphere/swarm/pull/1395.patch","issue_url":"https://api.github.com/repos/ethersphere/swarm/issues/1395","number":1395,"state":"open","locked":false,"title":"swarm/storage: Support for uploading 100gb files","user":{"login":"jmozah","id":940575,"node_id":"MDQ6VXNlcjk0MDU3NQ==","avatar_url":"https://avatars2.githubusercontent.com/u/940575?v=4","gravatar_id":"","url":"https://api.github.com/users/jmozah","html_url":"https://github.com/jmozah","followers_url":"https://api.github.com/users/jmozah/followers","following_url":"https://api.github.com/users/jmozah/following{/other_user}","gists_url":"https://api.github.com/users/jmozah/gists{/gist_id}","starred_url":"https://api.github.com/users/jmozah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmozah/subscriptions","organizations_url":"https://api.github.com/users/jmozah/orgs","repos_url":"https://api.github.com/users/jmozah/repos","events_url":"https://api.github.com/users/jmozah/events{/privacy}","received_events_url":"https://api.github.com/users/jmozah/received_events","type":"User","site_admin":false},"body":"Fixes #1356\r\n\r\nBefore:  Upload 100gb file fails\r\nAfter: 100gb upload passes in a 2 node cluster.\r\n\r\nsupersedes https://github.com/ethersphere/go-ethereum/pull/1357\r\n\r\n\r\n\r\n1) Go routine leaks:\r\n===============\r\n\r\nProblem: The go-routines spawned in the storeChunk() of \"hasherstore.go\" was not controlled. i.e. For every chunk a new go-routine was spawned leading to millions of go-routines when you upload a large file.\r\n\r\nFix: This was fixed by making the no of go-routines have a max limit and having a queue to help in the back pressure. So a max of 150 routines will be spawned (128 for data chunks in a branch + few for the tree chunks). Every routine will pick a chunk from the worker queue and push it to the storage. When all the chunks are over, they wait for the close signal from Wait() routine and die. The old contents of the Wait() function was moved to a new function called startWait() so that it can independently collect the chunk stats. We tried to use tags for doing the chunk stats, but unit testing became too challenging as stats are incremented all over the storage code.\r\n\r\n2) Chunk corruption when uploading file greater than 8.5gb in size\r\n=== ===============================================\r\n\r\nProblem: Whenever a file greater than 8.5Gb approx. was uploaded, then downloaded, the diffs would not match. This did not occur in smaller files.\r\n\r\nFix: The buildTree() function in pyramid.go is called every-time the data chunks reaches 128 or the file is over. This function is responsible for creating the tree chunk which binds the created data chunks in to the rest of the merkel tree. The first loop in this function determines how far the tree has wrapped. For example: if the current branch is 127th, then by adding a new tree chunk, you not only have to add the data chunks to this new tree chunk... but also since this is a chunk boundary, you go one level up. In some cases, when you add a final chunk, you close many levels on the top.\r\nEx: If level 0,1,2,3 has 127 branches, by adding a final tree chunk in level 0, level 0,1,2,3 all level becomes closed. This was not properly handles in the code. Now, when every tree chunk is created, we go back up to 128th level to see if this addition is going to trigger a snowball effect on the rest of the top levels.\r\n","created_at":"2019-05-14T14:54:34Z","updated_at":"2019-06-13T06:47:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b1c08fc821a9af66c97ac46eb01503add4b2268c","assignee":{"login":"jmozah","id":940575,"node_id":"MDQ6VXNlcjk0MDU3NQ==","avatar_url":"https://avatars2.githubusercontent.com/u/940575?v=4","gravatar_id":"","url":"https://api.github.com/users/jmozah","html_url":"https://github.com/jmozah","followers_url":"https://api.github.com/users/jmozah/followers","following_url":"https://api.github.com/users/jmozah/following{/other_user}","gists_url":"https://api.github.com/users/jmozah/gists{/gist_id}","starred_url":"https://api.github.com/users/jmozah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmozah/subscriptions","organizations_url":"https://api.github.com/users/jmozah/orgs","repos_url":"https://api.github.com/users/jmozah/repos","events_url":"https://api.github.com/users/jmozah/events{/privacy}","received_events_url":"https://api.github.com/users/jmozah/received_events","type":"User","site_admin":false},"assignees":[{"login":"jmozah","id":940575,"node_id":"MDQ6VXNlcjk0MDU3NQ==","avatar_url":"https://avatars2.githubusercontent.com/u/940575?v=4","gravatar_id":"","url":"https://api.github.com/users/jmozah","html_url":"https://github.com/jmozah","followers_url":"https://api.github.com/users/jmozah/followers","following_url":"https://api.github.com/users/jmozah/following{/other_user}","gists_url":"https://api.github.com/users/jmozah/gists{/gist_id}","starred_url":"https://api.github.com/users/jmozah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmozah/subscriptions","organizations_url":"https://api.github.com/users/jmozah/orgs","repos_url":"https://api.github.com/users/jmozah/repos","events_url":"https://api.github.com/users/jmozah/events{/privacy}","received_events_url":"https://api.github.com/users/jmozah/received_events","type":"User","site_admin":false},{"login":"acud","id":12988138,"node_id":"MDQ6VXNlcjEyOTg4MTM4","avatar_url":"https://avatars2.githubusercontent.com/u/12988138?v=4","gravatar_id":"","url":"https://api.github.com/users/acud","html_url":"https://github.com/acud","followers_url":"https://api.github.com/users/acud/followers","following_url":"https://api.github.com/users/acud/following{/other_user}","gists_url":"https://api.github.com/users/acud/gists{/gist_id}","starred_url":"https://api.github.com/users/acud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acud/subscriptions","organizations_url":"https://api.github.com/users/acud/orgs","repos_url":"https://api.github.com/users/acud/repos","events_url":"https://api.github.com/users/acud/events{/privacy}","received_events_url":"https://api.github.com/users/acud/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/ethersphere/swarm/pulls/1395/commits","review_comments_url":"https://api.github.com/repos/ethersphere/swarm/pulls/1395/comments","review_comment_url":"https://api.github.com/repos/ethersphere/swarm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ethersphere/swarm/issues/1395/comments","statuses_url":"https://api.github.com/repos/ethersphere/swarm/statuses/4c2cfe7bc3ece84c864291b2447c24c3e6c6f074","head":{"label":"fairDataSociety:master","ref":"master","sha":"4c2cfe7bc3ece84c864291b2447c24c3e6c6f074","user":{"login":"fairDataSociety","id":45981195,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ1OTgxMTk1","avatar_url":"https://avatars1.githubusercontent.com/u/45981195?v=4","gravatar_id":"","url":"https://api.github.com/users/fairDataSociety","html_url":"https://github.com/fairDataSociety","followers_url":"https://api.github.com/users/fairDataSociety/followers","following_url":"https://api.github.com/users/fairDataSociety/following{/other_user}","gists_url":"https://api.github.com/users/fairDataSociety/gists{/gist_id}","starred_url":"https://api.github.com/users/fairDataSociety/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fairDataSociety/subscriptions","organizations_url":"https://api.github.com/users/fairDataSociety/orgs","repos_url":"https://api.github.com/users/fairDataSociety/repos","events_url":"https://api.github.com/users/fairDataSociety/events{/privacy}","received_events_url":"https://api.github.com/users/fairDataSociety/received_events","type":"Organization","site_admin":false},"repo":{"id":183747181,"node_id":"MDEwOlJlcG9zaXRvcnkxODM3NDcxODE=","name":"swarm","full_name":"fairDataSociety/swarm","private":false,"owner":{"login":"fairDataSociety","id":45981195,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ1OTgxMTk1","avatar_url":"https://avatars1.githubusercontent.com/u/45981195?v=4","gravatar_id":"","url":"https://api.github.com/users/fairDataSociety","html_url":"https://github.com/fairDataSociety","followers_url":"https://api.github.com/users/fairDataSociety/followers","following_url":"https://api.github.com/users/fairDataSociety/following{/other_user}","gists_url":"https://api.github.com/users/fairDataSociety/gists{/gist_id}","starred_url":"https://api.github.com/users/fairDataSociety/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fairDataSociety/subscriptions","organizations_url":"https://api.github.com/users/fairDataSociety/orgs","repos_url":"https://api.github.com/users/fairDataSociety/repos","events_url":"https://api.github.com/users/fairDataSociety/events{/privacy}","received_events_url":"https://api.github.com/users/fairDataSociety/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/fairDataSociety/swarm","description":"Swarm: Censorship resistant storage and communication infrastructure for a sovereign digital society","fork":true,"url":"https://api.github.com/repos/fairDataSociety/swarm","forks_url":"https://api.github.com/repos/fairDataSociety/swarm/forks","keys_url":"https://api.github.com/repos/fairDataSociety/swarm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fairDataSociety/swarm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fairDataSociety/swarm/teams","hooks_url":"https://api.github.com/repos/fairDataSociety/swarm/hooks","issue_events_url":"https://api.github.com/repos/fairDataSociety/swarm/issues/events{/number}","events_url":"https://api.github.com/repos/fairDataSociety/swarm/events","assignees_url":"https://api.github.com/repos/fairDataSociety/swarm/assignees{/user}","branches_url":"https://api.github.com/repos/fairDataSociety/swarm/branches{/branch}","tags_url":"https://api.github.com/repos/fairDataSociety/swarm/tags","blobs_url":"https://api.github.com/repos/fairDataSociety/swarm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fairDataSociety/swarm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fairDataSociety/swarm/git/refs{/sha}","trees_url":"https://api.github.com/repos/fairDataSociety/swarm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fairDataSociety/swarm/statuses/{sha}","languages_url":"https://api.github.com/repos/fairDataSociety/swarm/languages","stargazers_url":"https://api.github.com/repos/fairDataSociety/swarm/stargazers","contributors_url":"https://api.github.com/repos/fairDataSociety/swarm/contributors","subscribers_url":"https://api.github.com/repos/fairDataSociety/swarm/subscribers","subscription_url":"https://api.github.com/repos/fairDataSociety/swarm/subscription","commits_url":"https://api.github.com/repos/fairDataSociety/swarm/commits{/sha}","git_commits_url":"https://api.github.com/repos/fairDataSociety/swarm/git/commits{/sha}","comments_url":"https://api.github.com/repos/fairDataSociety/swarm/comments{/number}","issue_comment_url":"https://api.github.com/repos/fairDataSociety/swarm/issues/comments{/number}","contents_url":"https://api.github.com/repos/fairDataSociety/swarm/contents/{+path}","compare_url":"https://api.github.com/repos/fairDataSociety/swarm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fairDataSociety/swarm/merges","archive_url":"https://api.github.com/repos/fairDataSociety/swarm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fairDataSociety/swarm/downloads","issues_url":"https://api.github.com/repos/fairDataSociety/swarm/issues{/number}","pulls_url":"https://api.github.com/repos/fairDataSociety/swarm/pulls{/number}","milestones_url":"https://api.github.com/repos/fairDataSociety/swarm/milestones{/number}","notifications_url":"https://api.github.com/repos/fairDataSociety/swarm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fairDataSociety/swarm/labels{/name}","releases_url":"https://api.github.com/repos/fairDataSociety/swarm/releases{/id}","deployments_url":"https://api.github.com/repos/fairDataSociety/swarm/deployments","created_at":"2019-04-27T08:21:29Z","updated_at":"2019-06-12T19:16:07Z","pushed_at":"2019-06-12T19:16:04Z","git_url":"git://github.com/fairDataSociety/swarm.git","ssh_url":"git@github.com:fairDataSociety/swarm.git","clone_url":"https://github.com/fairDataSociety/swarm.git","svn_url":"https://github.com/fairDataSociety/swarm","homepage":"https://swarm.ethereum.org/","size":124498,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":0,"license":{"key":"lgpl-3.0","name":"GNU Lesser General Public License v3.0","spdx_id":"LGPL-3.0","url":"https://api.github.com/licenses/lgpl-3.0","node_id":"MDc6TGljZW5zZTEy"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ethersphere:master","ref":"master","sha":"76e71cca3b7a63f0bc44fa9ac600fc86db1e15e7","user":{"login":"ethersphere","id":6946989,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NDY5ODk=","avatar_url":"https://avatars2.githubusercontent.com/u/6946989?v=4","gravatar_id":"","url":"https://api.github.com/users/ethersphere","html_url":"https://github.com/ethersphere","followers_url":"https://api.github.com/users/ethersphere/followers","following_url":"https://api.github.com/users/ethersphere/following{/other_user}","gists_url":"https://api.github.com/users/ethersphere/gists{/gist_id}","starred_url":"https://api.github.com/users/ethersphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethersphere/subscriptions","organizations_url":"https://api.github.com/users/ethersphere/orgs","repos_url":"https://api.github.com/users/ethersphere/repos","events_url":"https://api.github.com/users/ethersphere/events{/privacy}","received_events_url":"https://api.github.com/users/ethersphere/received_events","type":"Organization","site_admin":false},"repo":{"id":17790136,"node_id":"MDEwOlJlcG9zaXRvcnkxNzc5MDEzNg==","name":"swarm","full_name":"ethersphere/swarm","private":false,"owner":{"login":"ethersphere","id":6946989,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY5NDY5ODk=","avatar_url":"https://avatars2.githubusercontent.com/u/6946989?v=4","gravatar_id":"","url":"https://api.github.com/users/ethersphere","html_url":"https://github.com/ethersphere","followers_url":"https://api.github.com/users/ethersphere/followers","following_url":"https://api.github.com/users/ethersphere/following{/other_user}","gists_url":"https://api.github.com/users/ethersphere/gists{/gist_id}","starred_url":"https://api.github.com/users/ethersphere/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethersphere/subscriptions","organizations_url":"https://api.github.com/users/ethersphere/orgs","repos_url":"https://api.github.com/users/ethersphere/repos","events_url":"https://api.github.com/users/ethersphere/events{/privacy}","received_events_url":"https://api.github.com/users/ethersphere/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/ethersphere/swarm","description":"Swarm: Censorship resistant storage and communication infrastructure for a truly sovereign digital society","fork":false,"url":"https://api.github.com/repos/ethersphere/swarm","forks_url":"https://api.github.com/repos/ethersphere/swarm/forks","keys_url":"https://api.github.com/repos/ethersphere/swarm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ethersphere/swarm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ethersphere/swarm/teams","hooks_url":"https://api.github.com/repos/ethersphere/swarm/hooks","issue_events_url":"https://api.github.com/repos/ethersphere/swarm/issues/events{/number}","events_url":"https://api.github.com/repos/ethersphere/swarm/events","assignees_url":"https://api.github.com/repos/ethersphere/swarm/assignees{/user}","branches_url":"https://api.github.com/repos/ethersphere/swarm/branches{/branch}","tags_url":"https://api.github.com/repos/ethersphere/swarm/tags","blobs_url":"https://api.github.com/repos/ethersphere/swarm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ethersphere/swarm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ethersphere/swarm/git/refs{/sha}","trees_url":"https://api.github.com/repos/ethersphere/swarm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ethersphere/swarm/statuses/{sha}","languages_url":"https://api.github.com/repos/ethersphere/swarm/languages","stargazers_url":"https://api.github.com/repos/ethersphere/swarm/stargazers","contributors_url":"https://api.github.com/repos/ethersphere/swarm/contributors","subscribers_url":"https://api.github.com/repos/ethersphere/swarm/subscribers","subscription_url":"https://api.github.com/repos/ethersphere/swarm/subscription","commits_url":"https://api.github.com/repos/ethersphere/swarm/commits{/sha}","git_commits_url":"https://api.github.com/repos/ethersphere/swarm/git/commits{/sha}","comments_url":"https://api.github.com/repos/ethersphere/swarm/comments{/number}","issue_comment_url":"https://api.github.com/repos/ethersphere/swarm/issues/comments{/number}","contents_url":"https://api.github.com/repos/ethersphere/swarm/contents/{+path}","compare_url":"https://api.github.com/repos/ethersphere/swarm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ethersphere/swarm/merges","archive_url":"https://api.github.com/repos/ethersphere/swarm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ethersphere/swarm/downloads","issues_url":"https://api.github.com/repos/ethersphere/swarm/issues{/number}","pulls_url":"https://api.github.com/repos/ethersphere/swarm/pulls{/number}","milestones_url":"https://api.github.com/repos/ethersphere/swarm/milestones{/number}","notifications_url":"https://api.github.com/repos/ethersphere/swarm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ethersphere/swarm/labels{/name}","releases_url":"https://api.github.com/repos/ethersphere/swarm/releases{/id}","deployments_url":"https://api.github.com/repos/ethersphere/swarm/deployments","created_at":"2014-03-16T02:06:34Z","updated_at":"2019-06-12T13:50:48Z","pushed_at":"2019-06-12T19:16:06Z","git_url":"git://github.com/ethersphere/swarm.git","ssh_url":"git@github.com:ethersphere/swarm.git","clone_url":"https://github.com/ethersphere/swarm.git","svn_url":"https://github.com/ethersphere/swarm","homepage":"https://swarm.ethereum.org/","size":124184,"stargazers_count":139,"watchers_count":139,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":47,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":178,"license":{"key":"lgpl-3.0","name":"GNU Lesser General Public License v3.0","spdx_id":"LGPL-3.0","url":"https://api.github.com/licenses/lgpl-3.0","node_id":"MDc6TGljZW5zZTEy"},"forks":47,"open_issues":178,"watchers":139,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/1395"},"html":{"href":"https://github.com/ethersphere/swarm/pull/1395"},"issue":{"href":"https://api.github.com/repos/ethersphere/swarm/issues/1395"},"comments":{"href":"https://api.github.com/repos/ethersphere/swarm/issues/1395/comments"},"review_comments":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/1395/comments"},"review_comment":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ethersphere/swarm/pulls/1395/commits"},"statuses":{"href":"https://api.github.com/repos/ethersphere/swarm/statuses/4c2cfe7bc3ece84c864291b2447c24c3e6c6f074"}},"author_association":"NONE"}}
{"actor":{"display_login":"nolash"}}