{"action":"created","comment":{"url":"https://api.github.com/repos/scality/backbeat/pulls/comments/202499521","pull_request_review_id":137221791,"id":202499521,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwMjQ5OTUyMQ==","diff_hunk":"@@ -372,14 +334,30 @@ class MultipleBackendTask extends ReplicateObject {\n \n     /**\n      * Get byte ranges for an object of the given content length, such that the\n-     * range count does not exceed 1024 elements.\n+     * range count does not exceed 1024 parts if replicating to GCP or does not\n+     * exceed 10000 parts otherwise. This method also optimizes for the\n+     * subsquent range requests by returning ranges that are sized as powers of\n+     * two (aside from the last part).\n      * @param {Number} contentLen - The content length of the whole object\n+     * @param {Boolean} isGCP - Whether the object is being replicated to GCP\n      * @return {Array} The array of byte ranges.\n      */\n-    _getRanges(contentLen) {\n-        // Creates ranges that have sizes that are powers of 2.\n+    _getRanges(contentLen, isGCP) {\n+        // 5MB min part size rounded up to the nearest power of two is 8388608.\n+        let partCount = Math.pow(2,","path":"extensions/replication/tasks/MultipleBackendTask.js","position":87,"original_position":87,"commit_id":"951352398754d9fe41e1037f8d9fd6c0019e9d76","original_commit_id":"951352398754d9fe41e1037f8d9fd6c0019e9d76","user":{"login":"jonathan-gramain","id":3418241,"node_id":"MDQ6VXNlcjM0MTgyNDE=","avatar_url":"https://avatars1.githubusercontent.com/u/3418241?v=4","gravatar_id":"","url":"https://api.github.com/users/jonathan-gramain","html_url":"https://github.com/jonathan-gramain","followers_url":"https://api.github.com/users/jonathan-gramain/followers","following_url":"https://api.github.com/users/jonathan-gramain/following{/other_user}","gists_url":"https://api.github.com/users/jonathan-gramain/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathan-gramain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathan-gramain/subscriptions","organizations_url":"https://api.github.com/users/jonathan-gramain/orgs","repos_url":"https://api.github.com/users/jonathan-gramain/repos","events_url":"https://api.github.com/users/jonathan-gramain/events{/privacy}","received_events_url":"https://api.github.com/users/jonathan-gramain/received_events","type":"User","site_admin":false},"body":"Not sure what is the benefit of ensuring the number of parts is a power of two? Something that may benefit from being a power of two instead (or being divisible by a sufficiently large power-of-two number) could be the number of bytes per part, since a lot of layers in our systems tend to perform better when processing data aligned with disk or memory pages, generally speaking.\r\n\r\nUsually using a fixed default range size that can be optionally tuned works well in practice, and we should increase the range size if needed so that we stay under 10K ranges.","created_at":"2018-07-14T00:38:00Z","updated_at":"2018-07-14T00:51:29Z","html_url":"https://github.com/scality/backbeat/pull/381#discussion_r202499521","pull_request_url":"https://api.github.com/repos/scality/backbeat/pulls/381","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/scality/backbeat/pulls/comments/202499521"},"html":{"href":"https://github.com/scality/backbeat/pull/381#discussion_r202499521"},"pull_request":{"href":"https://api.github.com/repos/scality/backbeat/pulls/381"}},"in_reply_to_id":202488964},"pull_request":{"url":"https://api.github.com/repos/scality/backbeat/pulls/381","id":201140455,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAxMTQwNDU1","html_url":"https://github.com/scality/backbeat/pull/381","diff_url":"https://github.com/scality/backbeat/pull/381.diff","patch_url":"https://github.com/scality/backbeat/pull/381.patch","issue_url":"https://api.github.com/repos/scality/backbeat/issues/381","number":381,"state":"open","locked":false,"title":"bugfix: ZENKO-623 Use a ranged GET for MPU replication","user":{"login":"bennettbuchanan","id":15207067,"node_id":"MDQ6VXNlcjE1MjA3MDY3","avatar_url":"https://avatars0.githubusercontent.com/u/15207067?v=4","gravatar_id":"","url":"https://api.github.com/users/bennettbuchanan","html_url":"https://github.com/bennettbuchanan","followers_url":"https://api.github.com/users/bennettbuchanan/followers","following_url":"https://api.github.com/users/bennettbuchanan/following{/other_user}","gists_url":"https://api.github.com/users/bennettbuchanan/gists{/gist_id}","starred_url":"https://api.github.com/users/bennettbuchanan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bennettbuchanan/subscriptions","organizations_url":"https://api.github.com/users/bennettbuchanan/orgs","repos_url":"https://api.github.com/users/bennettbuchanan/repos","events_url":"https://api.github.com/users/bennettbuchanan/events{/privacy}","received_events_url":"https://api.github.com/users/bennettbuchanan/received_events","type":"User","site_admin":false},"body":"Use ranged gets for the source object instead of using a part number. A similar change needs to be applied for CRR to a zenko destination in a separate, refactor PR to follow.\r\n\r\nRationale:\r\n\r\nWhen the source object is stored in an external backend the location array does not include all the part information. Instead it stores the parts as a single element in the array (for example, [here](https://github.com/scality/S3/blob/development/8.0/lib/api/completeMultipartUpload.js#L211)):\r\n\r\n```\r\n[ { key: 'source-key',\r\n    start: 0,\r\n    size: 10485782,\r\n    dataStoreName: 'source-location',\r\n    dataStoreETag: 'dbff4c86d9f3daa594119662fb8e94b1-2' } ]\r\n```\r\n\r\nWhen this occurs, the multiple backend task attempts to retrieve the part number from the `dataStoreETag` value [here](https://github.com/scality/Arsenal/blob/development/8.0/lib/models/ObjectMDLocation.js#L46), but since the part numbers were never stored with the splitter `:`, we unintentionally send `NaN` as the part number resulting in the following logs:\r\n\r\n```\r\n{\"name\":\"Backbeat:Replication:QueueProcessor\",\"method\":\"MultipleBackendTask._putMPUPart\",\"entry\":{\"bucket\":\"mehmeh\",\"objectKey\":\"Transmit 5.0.5.zip\",\"versionId\":\"98469702552383999999RG001 23.1\",\"isDeleteMarker\":\"\"},\"origin\":\"source\",\"peer\":{\"host\":\"zenko-cloudserver-replicator\",\"port\":8000},\"error\":\"Part number must be a number.\",\"httpStatus\":400,\"time\":1530297452207,\"req_id\":\"ccdf12d7104ba92bdb98\",\"level\":\"error\",\"message\":\"an error occurred on getObject from S3\",\"hostname\":\"backbeat-consumer-5f768fd94c-n9jv9\",\"pid\":27}\r\n```\r\n\r\nWhile the object is replicated properly, this change has the affect that the destination ETag will differ from the source because the MPU parts are composed of different ranges.","created_at":"2018-07-12T22:15:16Z","updated_at":"2018-07-14T00:51:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"55444a7e4b0cf8c95f06c3fdb2c39f7a41cb7937","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/scality/backbeat/pulls/381/commits","review_comments_url":"https://api.github.com/repos/scality/backbeat/pulls/381/comments","review_comment_url":"https://api.github.com/repos/scality/backbeat/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scality/backbeat/issues/381/comments","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/951352398754d9fe41e1037f8d9fd6c0019e9d76","head":{"label":"scality:bugfix/ZENKO-623/use-range-calculation-for-all-MPUs","ref":"bugfix/ZENKO-623/use-range-calculation-for-all-MPUs","sha":"951352398754d9fe41e1037f8d9fd6c0019e9d76","user":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"repo":{"id":81391071,"node_id":"MDEwOlJlcG9zaXRvcnk4MTM5MTA3MQ==","name":"backbeat","full_name":"scality/backbeat","owner":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scality/backbeat","description":"Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.","fork":false,"url":"https://api.github.com/repos/scality/backbeat","forks_url":"https://api.github.com/repos/scality/backbeat/forks","keys_url":"https://api.github.com/repos/scality/backbeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scality/backbeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scality/backbeat/teams","hooks_url":"https://api.github.com/repos/scality/backbeat/hooks","issue_events_url":"https://api.github.com/repos/scality/backbeat/issues/events{/number}","events_url":"https://api.github.com/repos/scality/backbeat/events","assignees_url":"https://api.github.com/repos/scality/backbeat/assignees{/user}","branches_url":"https://api.github.com/repos/scality/backbeat/branches{/branch}","tags_url":"https://api.github.com/repos/scality/backbeat/tags","blobs_url":"https://api.github.com/repos/scality/backbeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scality/backbeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scality/backbeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/scality/backbeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/{sha}","languages_url":"https://api.github.com/repos/scality/backbeat/languages","stargazers_url":"https://api.github.com/repos/scality/backbeat/stargazers","contributors_url":"https://api.github.com/repos/scality/backbeat/contributors","subscribers_url":"https://api.github.com/repos/scality/backbeat/subscribers","subscription_url":"https://api.github.com/repos/scality/backbeat/subscription","commits_url":"https://api.github.com/repos/scality/backbeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/scality/backbeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/scality/backbeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/scality/backbeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/scality/backbeat/contents/{+path}","compare_url":"https://api.github.com/repos/scality/backbeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scality/backbeat/merges","archive_url":"https://api.github.com/repos/scality/backbeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scality/backbeat/downloads","issues_url":"https://api.github.com/repos/scality/backbeat/issues{/number}","pulls_url":"https://api.github.com/repos/scality/backbeat/pulls{/number}","milestones_url":"https://api.github.com/repos/scality/backbeat/milestones{/number}","notifications_url":"https://api.github.com/repos/scality/backbeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scality/backbeat/labels{/name}","releases_url":"https://api.github.com/repos/scality/backbeat/releases{/id}","deployments_url":"https://api.github.com/repos/scality/backbeat/deployments","created_at":"2017-02-09T00:31:22Z","updated_at":"2018-07-11T21:00:26Z","pushed_at":"2018-07-14T00:31:30Z","git_url":"git://github.com/scality/backbeat.git","ssh_url":"git@github.com:scality/backbeat.git","clone_url":"https://github.com/scality/backbeat.git","svn_url":"https://github.com/scality/backbeat","homepage":"https://www.zenko.io","size":2703,"stargazers_count":17,"watchers_count":17,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":12,"open_issues":17,"watchers":17,"default_branch":"development/8.0"}},"base":{"label":"scality:development/8.0","ref":"development/8.0","sha":"ac5528c854c69ab8c6bafc5f9c2a540300a3b7b4","user":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"repo":{"id":81391071,"node_id":"MDEwOlJlcG9zaXRvcnk4MTM5MTA3MQ==","name":"backbeat","full_name":"scality/backbeat","owner":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scality/backbeat","description":"Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.","fork":false,"url":"https://api.github.com/repos/scality/backbeat","forks_url":"https://api.github.com/repos/scality/backbeat/forks","keys_url":"https://api.github.com/repos/scality/backbeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scality/backbeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scality/backbeat/teams","hooks_url":"https://api.github.com/repos/scality/backbeat/hooks","issue_events_url":"https://api.github.com/repos/scality/backbeat/issues/events{/number}","events_url":"https://api.github.com/repos/scality/backbeat/events","assignees_url":"https://api.github.com/repos/scality/backbeat/assignees{/user}","branches_url":"https://api.github.com/repos/scality/backbeat/branches{/branch}","tags_url":"https://api.github.com/repos/scality/backbeat/tags","blobs_url":"https://api.github.com/repos/scality/backbeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scality/backbeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scality/backbeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/scality/backbeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/{sha}","languages_url":"https://api.github.com/repos/scality/backbeat/languages","stargazers_url":"https://api.github.com/repos/scality/backbeat/stargazers","contributors_url":"https://api.github.com/repos/scality/backbeat/contributors","subscribers_url":"https://api.github.com/repos/scality/backbeat/subscribers","subscription_url":"https://api.github.com/repos/scality/backbeat/subscription","commits_url":"https://api.github.com/repos/scality/backbeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/scality/backbeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/scality/backbeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/scality/backbeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/scality/backbeat/contents/{+path}","compare_url":"https://api.github.com/repos/scality/backbeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scality/backbeat/merges","archive_url":"https://api.github.com/repos/scality/backbeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scality/backbeat/downloads","issues_url":"https://api.github.com/repos/scality/backbeat/issues{/number}","pulls_url":"https://api.github.com/repos/scality/backbeat/pulls{/number}","milestones_url":"https://api.github.com/repos/scality/backbeat/milestones{/number}","notifications_url":"https://api.github.com/repos/scality/backbeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scality/backbeat/labels{/name}","releases_url":"https://api.github.com/repos/scality/backbeat/releases{/id}","deployments_url":"https://api.github.com/repos/scality/backbeat/deployments","created_at":"2017-02-09T00:31:22Z","updated_at":"2018-07-11T21:00:26Z","pushed_at":"2018-07-14T00:31:30Z","git_url":"git://github.com/scality/backbeat.git","ssh_url":"git@github.com:scality/backbeat.git","clone_url":"https://github.com/scality/backbeat.git","svn_url":"https://github.com/scality/backbeat","homepage":"https://www.zenko.io","size":2703,"stargazers_count":17,"watchers_count":17,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":12,"open_issues":17,"watchers":17,"default_branch":"development/8.0"}},"_links":{"self":{"href":"https://api.github.com/repos/scality/backbeat/pulls/381"},"html":{"href":"https://github.com/scality/backbeat/pull/381"},"issue":{"href":"https://api.github.com/repos/scality/backbeat/issues/381"},"comments":{"href":"https://api.github.com/repos/scality/backbeat/issues/381/comments"},"review_comments":{"href":"https://api.github.com/repos/scality/backbeat/pulls/381/comments"},"review_comment":{"href":"https://api.github.com/repos/scality/backbeat/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scality/backbeat/pulls/381/commits"},"statuses":{"href":"https://api.github.com/repos/scality/backbeat/statuses/951352398754d9fe41e1037f8d9fd6c0019e9d76"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jonathan-gramain"}}