{"action":"created","comment":{"url":"https://api.github.com/repos/kite-sdk/kite/pulls/comments/55218833","id":55218833,"diff_hunk":"@@ -60,4 +60,12 @@\n    * be closed and finalized once they reach this age.\n    */\n   public static final String ROLL_INTERVAL_S_PROP = \"kite.writer.roll-interval-seconds\";\n+\n+  /**\n+   * Until HADOOP-9565 is available and fully adopted, need to make this configurable so that\n+   * we can avoid multiple expensive copy operations when writing output to a file system that\n+   * does not support fast file renaming. Default is false unless the FileSystem URI scheme is\n+   * s3n or s3a.\n+   */\n+  public static final String OBJECTSTORE_FILESYSTEM = \"kite.writer.object-store\";","path":"kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/filesystem/FileSystemProperties.java","position":11,"original_position":11,"commit_id":"11e278f7b89be38e75854ec33c1b71331dacedc0","original_commit_id":"11e278f7b89be38e75854ec33c1b71331dacedc0","user":{"login":"rbrush","id":574824,"avatar_url":"https://avatars.githubusercontent.com/u/574824?v=3","gravatar_id":"","url":"https://api.github.com/users/rbrush","html_url":"https://github.com/rbrush","followers_url":"https://api.github.com/users/rbrush/followers","following_url":"https://api.github.com/users/rbrush/following{/other_user}","gists_url":"https://api.github.com/users/rbrush/gists{/gist_id}","starred_url":"https://api.github.com/users/rbrush/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rbrush/subscriptions","organizations_url":"https://api.github.com/users/rbrush/orgs","repos_url":"https://api.github.com/users/rbrush/repos","events_url":"https://api.github.com/users/rbrush/events{/privacy}","received_events_url":"https://api.github.com/users/rbrush/received_events","type":"User","site_admin":false},"body":"Just one suggestion here: it seems like this decision should be based on whether the filesystem supports efficient rename operations, since future object stores may support rename (and s3 may get it some day). Perhaps this setting should be \"kite.writer.supports-rename\", and by default we assume that s3a and s3n filesystems do not?\r\n\r\nI think this keeps Kite focused on the properties it's interested in...where it shouldn't really care or make assumptions about the properties of an object store.","created_at":"2016-03-07T15:29:25Z","updated_at":"2016-03-07T15:29:25Z","html_url":"https://github.com/kite-sdk/kite/pull/442#discussion_r55218833","pull_request_url":"https://api.github.com/repos/kite-sdk/kite/pulls/442","_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/comments/55218833"},"html":{"href":"https://github.com/kite-sdk/kite/pull/442#discussion_r55218833"},"pull_request":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/442"}}},"pull_request":{"url":"https://api.github.com/repos/kite-sdk/kite/pulls/442","id":61785252,"html_url":"https://github.com/kite-sdk/kite/pull/442","diff_url":"https://github.com/kite-sdk/kite/pull/442.diff","patch_url":"https://github.com/kite-sdk/kite/pull/442.patch","issue_url":"https://api.github.com/repos/kite-sdk/kite/issues/442","number":442,"state":"open","locked":false,"title":"KITE-1118: Optimize writing data to S3","user":{"login":"noslowerdna","id":930946,"avatar_url":"https://avatars.githubusercontent.com/u/930946?v=3","gravatar_id":"","url":"https://api.github.com/users/noslowerdna","html_url":"https://github.com/noslowerdna","followers_url":"https://api.github.com/users/noslowerdna/followers","following_url":"https://api.github.com/users/noslowerdna/following{/other_user}","gists_url":"https://api.github.com/users/noslowerdna/gists{/gist_id}","starred_url":"https://api.github.com/users/noslowerdna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noslowerdna/subscriptions","organizations_url":"https://api.github.com/users/noslowerdna/orgs","repos_url":"https://api.github.com/users/noslowerdna/repos","events_url":"https://api.github.com/users/noslowerdna/events{/privacy}","received_events_url":"https://api.github.com/users/noslowerdna/received_events","type":"User","site_admin":false},"body":"Pull request for https://issues.cloudera.org/browse/KITE-1118\r\n\r\nFor importing about 20 GB of data into S3, this provided approximately an 8x job speedup, from 40 min to 5 min.","created_at":"2016-03-04T23:09:04Z","updated_at":"2016-03-07T15:29:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cbae1265c614b10d050dfc5bd37b20657432ae4b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/kite-sdk/kite/pulls/442/commits","review_comments_url":"https://api.github.com/repos/kite-sdk/kite/pulls/442/comments","review_comment_url":"https://api.github.com/repos/kite-sdk/kite/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kite-sdk/kite/issues/442/comments","statuses_url":"https://api.github.com/repos/kite-sdk/kite/statuses/11e278f7b89be38e75854ec33c1b71331dacedc0","head":{"label":"noslowerdna:KITE-1118","ref":"KITE-1118","sha":"11e278f7b89be38e75854ec33c1b71331dacedc0","user":{"login":"noslowerdna","id":930946,"avatar_url":"https://avatars.githubusercontent.com/u/930946?v=3","gravatar_id":"","url":"https://api.github.com/users/noslowerdna","html_url":"https://github.com/noslowerdna","followers_url":"https://api.github.com/users/noslowerdna/followers","following_url":"https://api.github.com/users/noslowerdna/following{/other_user}","gists_url":"https://api.github.com/users/noslowerdna/gists{/gist_id}","starred_url":"https://api.github.com/users/noslowerdna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noslowerdna/subscriptions","organizations_url":"https://api.github.com/users/noslowerdna/orgs","repos_url":"https://api.github.com/users/noslowerdna/repos","events_url":"https://api.github.com/users/noslowerdna/events{/privacy}","received_events_url":"https://api.github.com/users/noslowerdna/received_events","type":"User","site_admin":false},"repo":{"id":23969382,"name":"kite","full_name":"noslowerdna/kite","owner":{"login":"noslowerdna","id":930946,"avatar_url":"https://avatars.githubusercontent.com/u/930946?v=3","gravatar_id":"","url":"https://api.github.com/users/noslowerdna","html_url":"https://github.com/noslowerdna","followers_url":"https://api.github.com/users/noslowerdna/followers","following_url":"https://api.github.com/users/noslowerdna/following{/other_user}","gists_url":"https://api.github.com/users/noslowerdna/gists{/gist_id}","starred_url":"https://api.github.com/users/noslowerdna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noslowerdna/subscriptions","organizations_url":"https://api.github.com/users/noslowerdna/orgs","repos_url":"https://api.github.com/users/noslowerdna/repos","events_url":"https://api.github.com/users/noslowerdna/events{/privacy}","received_events_url":"https://api.github.com/users/noslowerdna/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/noslowerdna/kite","description":"Kite SDK","fork":true,"url":"https://api.github.com/repos/noslowerdna/kite","forks_url":"https://api.github.com/repos/noslowerdna/kite/forks","keys_url":"https://api.github.com/repos/noslowerdna/kite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/noslowerdna/kite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/noslowerdna/kite/teams","hooks_url":"https://api.github.com/repos/noslowerdna/kite/hooks","issue_events_url":"https://api.github.com/repos/noslowerdna/kite/issues/events{/number}","events_url":"https://api.github.com/repos/noslowerdna/kite/events","assignees_url":"https://api.github.com/repos/noslowerdna/kite/assignees{/user}","branches_url":"https://api.github.com/repos/noslowerdna/kite/branches{/branch}","tags_url":"https://api.github.com/repos/noslowerdna/kite/tags","blobs_url":"https://api.github.com/repos/noslowerdna/kite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/noslowerdna/kite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/noslowerdna/kite/git/refs{/sha}","trees_url":"https://api.github.com/repos/noslowerdna/kite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/noslowerdna/kite/statuses/{sha}","languages_url":"https://api.github.com/repos/noslowerdna/kite/languages","stargazers_url":"https://api.github.com/repos/noslowerdna/kite/stargazers","contributors_url":"https://api.github.com/repos/noslowerdna/kite/contributors","subscribers_url":"https://api.github.com/repos/noslowerdna/kite/subscribers","subscription_url":"https://api.github.com/repos/noslowerdna/kite/subscription","commits_url":"https://api.github.com/repos/noslowerdna/kite/commits{/sha}","git_commits_url":"https://api.github.com/repos/noslowerdna/kite/git/commits{/sha}","comments_url":"https://api.github.com/repos/noslowerdna/kite/comments{/number}","issue_comment_url":"https://api.github.com/repos/noslowerdna/kite/issues/comments{/number}","contents_url":"https://api.github.com/repos/noslowerdna/kite/contents/{+path}","compare_url":"https://api.github.com/repos/noslowerdna/kite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/noslowerdna/kite/merges","archive_url":"https://api.github.com/repos/noslowerdna/kite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/noslowerdna/kite/downloads","issues_url":"https://api.github.com/repos/noslowerdna/kite/issues{/number}","pulls_url":"https://api.github.com/repos/noslowerdna/kite/pulls{/number}","milestones_url":"https://api.github.com/repos/noslowerdna/kite/milestones{/number}","notifications_url":"https://api.github.com/repos/noslowerdna/kite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/noslowerdna/kite/labels{/name}","releases_url":"https://api.github.com/repos/noslowerdna/kite/releases{/id}","deployments_url":"https://api.github.com/repos/noslowerdna/kite/deployments","created_at":"2014-09-12T17:08:22Z","updated_at":"2014-09-08T09:27:34Z","pushed_at":"2016-03-07T14:47:38Z","git_url":"git://github.com/noslowerdna/kite.git","ssh_url":"git@github.com:noslowerdna/kite.git","clone_url":"https://github.com/noslowerdna/kite.git","svn_url":"https://github.com/noslowerdna/kite","homepage":"http://kitesdk.org/docs/current/","size":40922,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kite-sdk:master","ref":"master","sha":"9e67becef9a0d9570d50f6727e1c12826bcfea28","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":14829187,"name":"kite","full_name":"kite-sdk/kite","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite","description":"Kite SDK","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite","forks_url":"https://api.github.com/repos/kite-sdk/kite/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite/issues/comments{/number}","contents_url":"https://api.github.com/repos/kite-sdk/kite/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite/releases{/id}","deployments_url":"https://api.github.com/repos/kite-sdk/kite/deployments","created_at":"2013-11-30T22:06:02Z","updated_at":"2016-02-25T13:39:27Z","pushed_at":"2016-03-07T14:47:38Z","git_url":"git://github.com/kite-sdk/kite.git","ssh_url":"git@github.com:kite-sdk/kite.git","clone_url":"https://github.com/kite-sdk/kite.git","svn_url":"https://github.com/kite-sdk/kite","homepage":"http://kitesdk.org/docs/current/","size":62495,"stargazers_count":196,"watchers_count":196,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":160,"mirror_url":null,"open_issues_count":32,"forks":160,"open_issues":32,"watchers":196,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/442"},"html":{"href":"https://github.com/kite-sdk/kite/pull/442"},"issue":{"href":"https://api.github.com/repos/kite-sdk/kite/issues/442"},"comments":{"href":"https://api.github.com/repos/kite-sdk/kite/issues/442/comments"},"review_comments":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/442/comments"},"review_comment":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kite-sdk/kite/pulls/442/commits"},"statuses":{"href":"https://api.github.com/repos/kite-sdk/kite/statuses/11e278f7b89be38e75854ec33c1b71331dacedc0"}}}}