{"action":"created","comment":{"url":"https://api.github.com/repos/scality/backbeat/pulls/comments/202834490","pull_request_review_id":137608971,"id":202834490,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwMjgzNDQ5MA==","diff_hunk":"@@ -0,0 +1,184 @@\n+# Transient CRR Source\n+\n+## Description\n+\n+The goal of this enhancement is to avoid the cost of reading data from\n+a public cloud one or more times for CRR purpose, when both the\n+primary and CRR targets are on public clouds.\n+\n+It may also be used for low-latency writes to local storage before\n+having CRR transitioning the data asynchronously to a cloud target.\n+\n+## Requirements\n+\n+* It must be possible to configure object locations and CRR targets to\n+  store objects on multiple public clouds (aka. external targets)\n+  without a permanent local storage target, while being able to have\n+  backbeat read from a local, non-cloud source for CRR purpose,\n+  avoiding the cost of reading from public clouds.\n+\n+* It must be possible to set a limit on the storage space consumed by\n+  a transient location, and returning errors to PUT requests if the\n+  limit is reached.\n+\n+## Design\n+\n+### Design overview\n+\n+CRR process reads from the source defined by the location constraint,\n+writes to one or more destinations asynchronously and updates statuses\n+to *COMPLETED* on the source when done.\n+\n+Transient source relies on this core behavior, but refers to a\n+temporary primary location for a new object, until the CRR completes\n+the processing of this object.\n+\n+The following then happens when an object is put to a location which\n+is configured as being a transient source:\n+\n+* the object is first put to the location which is configured as a\n+  transient source, then the put status is returned as usual,\n+\n+* CRR processors transfer the object asynchronously to all CRR targets\n+  set through replication API. It sends an extra header in the GET\n+  request \"x-amz-location-constraint\" to ensure reads are done from\n+  the transient source location, not from the *preferred read\n+  location*,\n+\n+* finally, once all CRR targets have been processed, the transient\n+  data is deleted asynchronously by the garbage collector backbeat\n+  service. Reads from users still happen on the *preferred read\n+  location*.\n+\n+If for some reason the CRR cannot be completed to some backends, the\n+data will be held on the transient source, and the data will still be\n+read from the *preferred read location* if CRR was successful on\n+it. If CRR is either PENDING or FAILED to the preferred read target,\n+it's still possible to provide the \"x-amz-location-constraint\" header\n+on GET requests, specifying the location from which to read the data.\n+\n+Once CRR has eventually completed successfully to all backends (either\n+through automatic or manually-triggered retries), the transient source\n+data is garbage-collected asynchronously, and the object will not be\n+readable anymore from the transient source. We do not update object\n+metadata to reflect the fact that the data on transient source has\n+effectively been deleted, the user must be aware that only the CRR\n+cloud targets remain valid sources.\n+\n+### Detailed design\n+\n+Here are the changes per component:\n+\n+#### CloudServer\n+\n+* support for \"isTransient\" property on locations: enforces that\n+  replication configuration of buckets that belong to transient\n+  locations specify a *preferred read location*\n+\n+* new backbeat route `/_/backbeat/batchDelete`, that is used by the\n+  garbage collector process to delete data locations. It is used\n+  through POST requests containing a JSON body that is an array of\n+  locations to delete.\n+\n+* modified support of *preferred read locations*: now it's set through\n+  the replication configuration of a bucket, by appending the\n+  `:preferred_read` suffix to the given location in the\n+  `<StorageClass>` attribute. This has support from the Orbit user\n+  interface through the locations setup.\n+\n+* added storage limit support through the use of Redis keys storing","path":"docs/transient-crr-source.md","position":89,"original_position":89,"commit_id":"e2e4ad59fe6520a304c6a7ae6d0cc28fedbfa04c","original_commit_id":"d5c05b6d7f4e1e12f5565594721f2dda0c107403","user":{"login":"jonathan-gramain","id":3418241,"node_id":"MDQ6VXNlcjM0MTgyNDE=","avatar_url":"https://avatars1.githubusercontent.com/u/3418241?v=4","gravatar_id":"","url":"https://api.github.com/users/jonathan-gramain","html_url":"https://github.com/jonathan-gramain","followers_url":"https://api.github.com/users/jonathan-gramain/followers","following_url":"https://api.github.com/users/jonathan-gramain/following{/other_user}","gists_url":"https://api.github.com/users/jonathan-gramain/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathan-gramain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathan-gramain/subscriptions","organizations_url":"https://api.github.com/users/jonathan-gramain/orgs","repos_url":"https://api.github.com/users/jonathan-gramain/repos","events_url":"https://api.github.com/users/jonathan-gramain/events{/privacy}","received_events_url":"https://api.github.com/users/jonathan-gramain/received_events","type":"User","site_admin":false},"body":"They are never removed right now.\r\n\r\nDora raised an interesting point that if we re-create a location later with the same name, we would re-use the same keys. Maybe this is a strong incentive to cleanup when we delete locations, or alternatively making sure the keys do not conflict when re-creating locations (she should be opening a ticket for this shortly).","created_at":"2018-07-16T21:44:53Z","updated_at":"2018-07-16T21:44:53Z","html_url":"https://github.com/scality/backbeat/pull/379#discussion_r202834490","pull_request_url":"https://api.github.com/repos/scality/backbeat/pulls/379","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/scality/backbeat/pulls/comments/202834490"},"html":{"href":"https://github.com/scality/backbeat/pull/379#discussion_r202834490"},"pull_request":{"href":"https://api.github.com/repos/scality/backbeat/pulls/379"}},"in_reply_to_id":202790424},"pull_request":{"url":"https://api.github.com/repos/scality/backbeat/pulls/379","id":200853548,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAwODUzNTQ4","html_url":"https://github.com/scality/backbeat/pull/379","diff_url":"https://github.com/scality/backbeat/pull/379.diff","patch_url":"https://github.com/scality/backbeat/pull/379.patch","issue_url":"https://api.github.com/repos/scality/backbeat/issues/379","number":379,"state":"open","locked":false,"title":"doc: transient CRR source","user":{"login":"jonathan-gramain","id":3418241,"node_id":"MDQ6VXNlcjM0MTgyNDE=","avatar_url":"https://avatars1.githubusercontent.com/u/3418241?v=4","gravatar_id":"","url":"https://api.github.com/users/jonathan-gramain","html_url":"https://github.com/jonathan-gramain","followers_url":"https://api.github.com/users/jonathan-gramain/followers","following_url":"https://api.github.com/users/jonathan-gramain/following{/other_user}","gists_url":"https://api.github.com/users/jonathan-gramain/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathan-gramain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathan-gramain/subscriptions","organizations_url":"https://api.github.com/users/jonathan-gramain/orgs","repos_url":"https://api.github.com/users/jonathan-gramain/repos","events_url":"https://api.github.com/users/jonathan-gramain/events{/privacy}","received_events_url":"https://api.github.com/users/jonathan-gramain/received_events","type":"User","site_admin":false},"body":"Replaces https://github.com/scality/backbeat/pull/212.","created_at":"2018-07-11T23:38:47Z","updated_at":"2018-07-16T21:44:53Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7c30ba8648fd4e5ccbf013a3c264f3b37ce5af19","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/scality/backbeat/pulls/379/commits","review_comments_url":"https://api.github.com/repos/scality/backbeat/pulls/379/comments","review_comment_url":"https://api.github.com/repos/scality/backbeat/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scality/backbeat/issues/379/comments","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/e2e4ad59fe6520a304c6a7ae6d0cc28fedbfa04c","head":{"label":"scality:feature/ZENKO-143-transientSourceDesignDoc","ref":"feature/ZENKO-143-transientSourceDesignDoc","sha":"e2e4ad59fe6520a304c6a7ae6d0cc28fedbfa04c","user":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"repo":{"id":81391071,"node_id":"MDEwOlJlcG9zaXRvcnk4MTM5MTA3MQ==","name":"backbeat","full_name":"scality/backbeat","owner":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scality/backbeat","description":"Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.","fork":false,"url":"https://api.github.com/repos/scality/backbeat","forks_url":"https://api.github.com/repos/scality/backbeat/forks","keys_url":"https://api.github.com/repos/scality/backbeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scality/backbeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scality/backbeat/teams","hooks_url":"https://api.github.com/repos/scality/backbeat/hooks","issue_events_url":"https://api.github.com/repos/scality/backbeat/issues/events{/number}","events_url":"https://api.github.com/repos/scality/backbeat/events","assignees_url":"https://api.github.com/repos/scality/backbeat/assignees{/user}","branches_url":"https://api.github.com/repos/scality/backbeat/branches{/branch}","tags_url":"https://api.github.com/repos/scality/backbeat/tags","blobs_url":"https://api.github.com/repos/scality/backbeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scality/backbeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scality/backbeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/scality/backbeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/{sha}","languages_url":"https://api.github.com/repos/scality/backbeat/languages","stargazers_url":"https://api.github.com/repos/scality/backbeat/stargazers","contributors_url":"https://api.github.com/repos/scality/backbeat/contributors","subscribers_url":"https://api.github.com/repos/scality/backbeat/subscribers","subscription_url":"https://api.github.com/repos/scality/backbeat/subscription","commits_url":"https://api.github.com/repos/scality/backbeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/scality/backbeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/scality/backbeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/scality/backbeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/scality/backbeat/contents/{+path}","compare_url":"https://api.github.com/repos/scality/backbeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scality/backbeat/merges","archive_url":"https://api.github.com/repos/scality/backbeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scality/backbeat/downloads","issues_url":"https://api.github.com/repos/scality/backbeat/issues{/number}","pulls_url":"https://api.github.com/repos/scality/backbeat/pulls{/number}","milestones_url":"https://api.github.com/repos/scality/backbeat/milestones{/number}","notifications_url":"https://api.github.com/repos/scality/backbeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scality/backbeat/labels{/name}","releases_url":"https://api.github.com/repos/scality/backbeat/releases{/id}","deployments_url":"https://api.github.com/repos/scality/backbeat/deployments","created_at":"2017-02-09T00:31:22Z","updated_at":"2018-07-11T21:00:26Z","pushed_at":"2018-07-16T21:17:32Z","git_url":"git://github.com/scality/backbeat.git","ssh_url":"git@github.com:scality/backbeat.git","clone_url":"https://github.com/scality/backbeat.git","svn_url":"https://github.com/scality/backbeat","homepage":"https://www.zenko.io","size":2703,"stargazers_count":17,"watchers_count":17,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":12,"open_issues":17,"watchers":17,"default_branch":"development/8.0"}},"base":{"label":"scality:development/8.0","ref":"development/8.0","sha":"ac5528c854c69ab8c6bafc5f9c2a540300a3b7b4","user":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"repo":{"id":81391071,"node_id":"MDEwOlJlcG9zaXRvcnk4MTM5MTA3MQ==","name":"backbeat","full_name":"scality/backbeat","owner":{"login":"scality","id":383812,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM4MzgxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/383812?v=4","gravatar_id":"","url":"https://api.github.com/users/scality","html_url":"https://github.com/scality","followers_url":"https://api.github.com/users/scality/followers","following_url":"https://api.github.com/users/scality/following{/other_user}","gists_url":"https://api.github.com/users/scality/gists{/gist_id}","starred_url":"https://api.github.com/users/scality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scality/subscriptions","organizations_url":"https://api.github.com/users/scality/orgs","repos_url":"https://api.github.com/users/scality/repos","events_url":"https://api.github.com/users/scality/events{/privacy}","received_events_url":"https://api.github.com/users/scality/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scality/backbeat","description":"Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.","fork":false,"url":"https://api.github.com/repos/scality/backbeat","forks_url":"https://api.github.com/repos/scality/backbeat/forks","keys_url":"https://api.github.com/repos/scality/backbeat/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scality/backbeat/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scality/backbeat/teams","hooks_url":"https://api.github.com/repos/scality/backbeat/hooks","issue_events_url":"https://api.github.com/repos/scality/backbeat/issues/events{/number}","events_url":"https://api.github.com/repos/scality/backbeat/events","assignees_url":"https://api.github.com/repos/scality/backbeat/assignees{/user}","branches_url":"https://api.github.com/repos/scality/backbeat/branches{/branch}","tags_url":"https://api.github.com/repos/scality/backbeat/tags","blobs_url":"https://api.github.com/repos/scality/backbeat/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scality/backbeat/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scality/backbeat/git/refs{/sha}","trees_url":"https://api.github.com/repos/scality/backbeat/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scality/backbeat/statuses/{sha}","languages_url":"https://api.github.com/repos/scality/backbeat/languages","stargazers_url":"https://api.github.com/repos/scality/backbeat/stargazers","contributors_url":"https://api.github.com/repos/scality/backbeat/contributors","subscribers_url":"https://api.github.com/repos/scality/backbeat/subscribers","subscription_url":"https://api.github.com/repos/scality/backbeat/subscription","commits_url":"https://api.github.com/repos/scality/backbeat/commits{/sha}","git_commits_url":"https://api.github.com/repos/scality/backbeat/git/commits{/sha}","comments_url":"https://api.github.com/repos/scality/backbeat/comments{/number}","issue_comment_url":"https://api.github.com/repos/scality/backbeat/issues/comments{/number}","contents_url":"https://api.github.com/repos/scality/backbeat/contents/{+path}","compare_url":"https://api.github.com/repos/scality/backbeat/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scality/backbeat/merges","archive_url":"https://api.github.com/repos/scality/backbeat/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scality/backbeat/downloads","issues_url":"https://api.github.com/repos/scality/backbeat/issues{/number}","pulls_url":"https://api.github.com/repos/scality/backbeat/pulls{/number}","milestones_url":"https://api.github.com/repos/scality/backbeat/milestones{/number}","notifications_url":"https://api.github.com/repos/scality/backbeat/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scality/backbeat/labels{/name}","releases_url":"https://api.github.com/repos/scality/backbeat/releases{/id}","deployments_url":"https://api.github.com/repos/scality/backbeat/deployments","created_at":"2017-02-09T00:31:22Z","updated_at":"2018-07-11T21:00:26Z","pushed_at":"2018-07-16T21:17:32Z","git_url":"git://github.com/scality/backbeat.git","ssh_url":"git@github.com:scality/backbeat.git","clone_url":"https://github.com/scality/backbeat.git","svn_url":"https://github.com/scality/backbeat","homepage":"https://www.zenko.io","size":2703,"stargazers_count":17,"watchers_count":17,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":17,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":12,"open_issues":17,"watchers":17,"default_branch":"development/8.0"}},"_links":{"self":{"href":"https://api.github.com/repos/scality/backbeat/pulls/379"},"html":{"href":"https://github.com/scality/backbeat/pull/379"},"issue":{"href":"https://api.github.com/repos/scality/backbeat/issues/379"},"comments":{"href":"https://api.github.com/repos/scality/backbeat/issues/379/comments"},"review_comments":{"href":"https://api.github.com/repos/scality/backbeat/pulls/379/comments"},"review_comment":{"href":"https://api.github.com/repos/scality/backbeat/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scality/backbeat/pulls/379/commits"},"statuses":{"href":"https://api.github.com/repos/scality/backbeat/statuses/e2e4ad59fe6520a304c6a7ae6d0cc28fedbfa04c"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jonathan-gramain"}}