{"action":"created","comment":{"url":"https://api.github.com/repos/apache/beam/pulls/comments/194556385","pull_request_review_id":127751730,"id":194556385,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NDU1NjM4NQ==","diff_hunk":"@@ -0,0 +1,96 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one\n+ * or more contributor license agreements.  See the NOTICE file\n+ * distributed with this work for additional information\n+ * regarding copyright ownership.  The ASF licenses this file\n+ * to you under the Apache License, Version 2.0 (the\n+ * \"License\"); you may not use this file except in compliance\n+ * with the License.  You may obtain a copy of the License at\n+ *\n+ *     http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+\n+package org.apache.beam.runners.fnexecution.artifact;\n+\n+\n+import com.google.protobuf.ByteString;\n+import io.grpc.stub.StreamObserver;\n+import java.io.InputStream;\n+import java.nio.ByteBuffer;\n+import java.nio.channels.Channels;\n+import java.nio.channels.ReadableByteChannel;\n+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;\n+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;\n+import org.apache.beam.sdk.io.FileSystems;\n+import org.apache.beam.sdk.io.fs.ResolveOptions;\n+import org.apache.beam.sdk.io.fs.ResourceId;\n+\n+/**\n+ * An {@link ArtifactRetrievalService} that uses distributed file systems as its backing storage.","path":"runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java","position":35,"original_position":35,"commit_id":"533ea7e5cccdb5225655372224767dbc8c0469e7","original_commit_id":"d9306e2d446b4707f55a3e6821f8f3d08f5e5ca6","user":{"login":"angoenka","id":38219,"node_id":"MDQ6VXNlcjM4MjE5","avatar_url":"https://avatars3.githubusercontent.com/u/38219?v=4","gravatar_id":"","url":"https://api.github.com/users/angoenka","html_url":"https://github.com/angoenka","followers_url":"https://api.github.com/users/angoenka/followers","following_url":"https://api.github.com/users/angoenka/following{/other_user}","gists_url":"https://api.github.com/users/angoenka/gists{/gist_id}","starred_url":"https://api.github.com/users/angoenka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angoenka/subscriptions","organizations_url":"https://api.github.com/users/angoenka/orgs","repos_url":"https://api.github.com/users/angoenka/repos","events_url":"https://api.github.com/users/angoenka/events{/privacy}","received_events_url":"https://api.github.com/users/angoenka/received_events","type":"User","site_admin":false},"body":"Artifact location can never be assumed to be $token/$artifactName.\r\n\r\nThe artifacts are saved with a different name than the provided artifact name and the mapping of artifact name vs file name is stored in the manifest file.\r\nThe ProxyManifest is the proto which describe the format of manifest file. Its an optional format but it fits well in our use case.\r\n\r\nBy providing the base dir name as the token, the retrieval service assumes the folder structure (what i mean is it assumes that the manifest will be  $stagingToken/MANIFEST) As we only need manifest file to get locations for all the artifacts, passing the path to manifest file as token will remove implicit directory structure and simplify the code.","created_at":"2018-06-11T21:38:25Z","updated_at":"2018-06-11T21:38:25Z","html_url":"https://github.com/apache/beam/pull/5584#discussion_r194556385","pull_request_url":"https://api.github.com/repos/apache/beam/pulls/5584","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/beam/pulls/comments/194556385"},"html":{"href":"https://github.com/apache/beam/pull/5584#discussion_r194556385"},"pull_request":{"href":"https://api.github.com/repos/apache/beam/pulls/5584"}},"in_reply_to_id":193914606},"pull_request":{"url":"https://api.github.com/repos/apache/beam/pulls/5584","id":193462487,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkzNDYyNDg3","html_url":"https://github.com/apache/beam/pull/5584","diff_url":"https://github.com/apache/beam/pull/5584.diff","patch_url":"https://github.com/apache/beam/pull/5584.patch","issue_url":"https://api.github.com/repos/apache/beam/issues/5584","number":5584,"state":"open","locked":false,"title":"[BEAM-4291] Add distributed artifact retrieval","user":{"login":"axelmagn","id":1660665,"node_id":"MDQ6VXNlcjE2NjA2NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/1660665?v=4","gravatar_id":"","url":"https://api.github.com/users/axelmagn","html_url":"https://github.com/axelmagn","followers_url":"https://api.github.com/users/axelmagn/followers","following_url":"https://api.github.com/users/axelmagn/following{/other_user}","gists_url":"https://api.github.com/users/axelmagn/gists{/gist_id}","starred_url":"https://api.github.com/users/axelmagn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/axelmagn/subscriptions","organizations_url":"https://api.github.com/users/axelmagn/orgs","repos_url":"https://api.github.com/users/axelmagn/repos","events_url":"https://api.github.com/users/axelmagn/events{/privacy}","received_events_url":"https://api.github.com/users/axelmagn/received_events","type":"User","site_admin":false},"body":"\r\nImplement an ArtifactRetrievalService that uses the cloud storage\r\nsolutions supported by sdk.io.FileSystems.\r\n\r\n------------------------\r\n\r\nFollow this checklist to help us incorporate your contribution quickly and easily:\r\n\r\n - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.\r\n - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).\r\n\r\nIt will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.\r\n","created_at":"2018-06-07T22:17:18Z","updated_at":"2018-06-11T21:38:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ef0c05d48813d8eff1c326c18a0d895ba9b788ec","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/beam/pulls/5584/commits","review_comments_url":"https://api.github.com/repos/apache/beam/pulls/5584/comments","review_comment_url":"https://api.github.com/repos/apache/beam/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/beam/issues/5584/comments","statuses_url":"https://api.github.com/repos/apache/beam/statuses/533ea7e5cccdb5225655372224767dbc8c0469e7","head":{"label":"axelmagn:dfs-artifact-retrieval","ref":"dfs-artifact-retrieval","sha":"533ea7e5cccdb5225655372224767dbc8c0469e7","user":{"login":"axelmagn","id":1660665,"node_id":"MDQ6VXNlcjE2NjA2NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/1660665?v=4","gravatar_id":"","url":"https://api.github.com/users/axelmagn","html_url":"https://github.com/axelmagn","followers_url":"https://api.github.com/users/axelmagn/followers","following_url":"https://api.github.com/users/axelmagn/following{/other_user}","gists_url":"https://api.github.com/users/axelmagn/gists{/gist_id}","starred_url":"https://api.github.com/users/axelmagn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/axelmagn/subscriptions","organizations_url":"https://api.github.com/users/axelmagn/orgs","repos_url":"https://api.github.com/users/axelmagn/repos","events_url":"https://api.github.com/users/axelmagn/events{/privacy}","received_events_url":"https://api.github.com/users/axelmagn/received_events","type":"User","site_admin":false},"repo":{"id":98220341,"node_id":"MDEwOlJlcG9zaXRvcnk5ODIyMDM0MQ==","name":"beam","full_name":"axelmagn/beam","owner":{"login":"axelmagn","id":1660665,"node_id":"MDQ6VXNlcjE2NjA2NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/1660665?v=4","gravatar_id":"","url":"https://api.github.com/users/axelmagn","html_url":"https://github.com/axelmagn","followers_url":"https://api.github.com/users/axelmagn/followers","following_url":"https://api.github.com/users/axelmagn/following{/other_user}","gists_url":"https://api.github.com/users/axelmagn/gists{/gist_id}","starred_url":"https://api.github.com/users/axelmagn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/axelmagn/subscriptions","organizations_url":"https://api.github.com/users/axelmagn/orgs","repos_url":"https://api.github.com/users/axelmagn/repos","events_url":"https://api.github.com/users/axelmagn/events{/privacy}","received_events_url":"https://api.github.com/users/axelmagn/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/axelmagn/beam","description":"Mirror of Apache Beam","fork":true,"url":"https://api.github.com/repos/axelmagn/beam","forks_url":"https://api.github.com/repos/axelmagn/beam/forks","keys_url":"https://api.github.com/repos/axelmagn/beam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/axelmagn/beam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/axelmagn/beam/teams","hooks_url":"https://api.github.com/repos/axelmagn/beam/hooks","issue_events_url":"https://api.github.com/repos/axelmagn/beam/issues/events{/number}","events_url":"https://api.github.com/repos/axelmagn/beam/events","assignees_url":"https://api.github.com/repos/axelmagn/beam/assignees{/user}","branches_url":"https://api.github.com/repos/axelmagn/beam/branches{/branch}","tags_url":"https://api.github.com/repos/axelmagn/beam/tags","blobs_url":"https://api.github.com/repos/axelmagn/beam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/axelmagn/beam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/axelmagn/beam/git/refs{/sha}","trees_url":"https://api.github.com/repos/axelmagn/beam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/axelmagn/beam/statuses/{sha}","languages_url":"https://api.github.com/repos/axelmagn/beam/languages","stargazers_url":"https://api.github.com/repos/axelmagn/beam/stargazers","contributors_url":"https://api.github.com/repos/axelmagn/beam/contributors","subscribers_url":"https://api.github.com/repos/axelmagn/beam/subscribers","subscription_url":"https://api.github.com/repos/axelmagn/beam/subscription","commits_url":"https://api.github.com/repos/axelmagn/beam/commits{/sha}","git_commits_url":"https://api.github.com/repos/axelmagn/beam/git/commits{/sha}","comments_url":"https://api.github.com/repos/axelmagn/beam/comments{/number}","issue_comment_url":"https://api.github.com/repos/axelmagn/beam/issues/comments{/number}","contents_url":"https://api.github.com/repos/axelmagn/beam/contents/{+path}","compare_url":"https://api.github.com/repos/axelmagn/beam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/axelmagn/beam/merges","archive_url":"https://api.github.com/repos/axelmagn/beam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/axelmagn/beam/downloads","issues_url":"https://api.github.com/repos/axelmagn/beam/issues{/number}","pulls_url":"https://api.github.com/repos/axelmagn/beam/pulls{/number}","milestones_url":"https://api.github.com/repos/axelmagn/beam/milestones{/number}","notifications_url":"https://api.github.com/repos/axelmagn/beam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/axelmagn/beam/labels{/name}","releases_url":"https://api.github.com/repos/axelmagn/beam/releases{/id}","deployments_url":"https://api.github.com/repos/axelmagn/beam/deployments","created_at":"2017-07-24T18:08:35Z","updated_at":"2018-06-11T20:49:25Z","pushed_at":"2018-06-11T21:26:03Z","git_url":"git://github.com/axelmagn/beam.git","ssh_url":"git@github.com:axelmagn/beam.git","clone_url":"https://github.com/axelmagn/beam.git","svn_url":"https://github.com/axelmagn/beam","homepage":"","size":60751,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"d906270f243bb4de20a7f0baf514667590c8c494","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":50904245,"node_id":"MDEwOlJlcG9zaXRvcnk1MDkwNDI0NQ==","name":"beam","full_name":"apache/beam","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/beam","description":"Apache Beam","fork":false,"url":"https://api.github.com/repos/apache/beam","forks_url":"https://api.github.com/repos/apache/beam/forks","keys_url":"https://api.github.com/repos/apache/beam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/beam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/beam/teams","hooks_url":"https://api.github.com/repos/apache/beam/hooks","issue_events_url":"https://api.github.com/repos/apache/beam/issues/events{/number}","events_url":"https://api.github.com/repos/apache/beam/events","assignees_url":"https://api.github.com/repos/apache/beam/assignees{/user}","branches_url":"https://api.github.com/repos/apache/beam/branches{/branch}","tags_url":"https://api.github.com/repos/apache/beam/tags","blobs_url":"https://api.github.com/repos/apache/beam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/beam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/beam/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/beam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/beam/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/beam/languages","stargazers_url":"https://api.github.com/repos/apache/beam/stargazers","contributors_url":"https://api.github.com/repos/apache/beam/contributors","subscribers_url":"https://api.github.com/repos/apache/beam/subscribers","subscription_url":"https://api.github.com/repos/apache/beam/subscription","commits_url":"https://api.github.com/repos/apache/beam/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/beam/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/beam/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/beam/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/beam/contents/{+path}","compare_url":"https://api.github.com/repos/apache/beam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/beam/merges","archive_url":"https://api.github.com/repos/apache/beam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/beam/downloads","issues_url":"https://api.github.com/repos/apache/beam/issues{/number}","pulls_url":"https://api.github.com/repos/apache/beam/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/beam/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/beam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/beam/labels{/name}","releases_url":"https://api.github.com/repos/apache/beam/releases{/id}","deployments_url":"https://api.github.com/repos/apache/beam/deployments","created_at":"2016-02-02T08:00:06Z","updated_at":"2018-06-11T21:19:43Z","pushed_at":"2018-06-11T21:29:03Z","git_url":"git://github.com/apache/beam.git","ssh_url":"git@github.com:apache/beam.git","clone_url":"https://github.com/apache/beam.git","svn_url":"https://github.com/apache/beam","homepage":"","size":65088,"stargazers_count":1912,"watchers_count":1912,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1183,"mirror_url":null,"archived":false,"open_issues_count":108,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1183,"open_issues":108,"watchers":1912,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/beam/pulls/5584"},"html":{"href":"https://github.com/apache/beam/pull/5584"},"issue":{"href":"https://api.github.com/repos/apache/beam/issues/5584"},"comments":{"href":"https://api.github.com/repos/apache/beam/issues/5584/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/beam/pulls/5584/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/beam/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/beam/pulls/5584/commits"},"statuses":{"href":"https://api.github.com/repos/apache/beam/statuses/533ea7e5cccdb5225655372224767dbc8c0469e7"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"angoenka"}}