{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/87072977","pull_request_review_id":7700071,"id":87072977,"diff_hunk":"@@ -86,6 +86,16 @@ abstract class FileCommitProtocol {\n   def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String], ext: String): String\n \n   /**\n+   * Similar to newTaskTempFile(), but allows files to committed to an absolute output location.\n+   * Depending on the implementation, there may be weaker guarantees around adding files this way.\n+   */\n+  def newTaskTempFileAbsPath(","path":"core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala","position":7,"original_position":7,"commit_id":"b5bbb1d664dd23131635c156b6b8628184e0cc4f","original_commit_id":"b5bbb1d664dd23131635c156b6b8628184e0cc4f","user":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"body":"@rxin","created_at":"2016-11-08T20:09:19Z","updated_at":"2016-11-08T20:09:19Z","html_url":"https://github.com/apache/spark/pull/15814#discussion_r87072977","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/15814","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/87072977"},"html":{"href":"https://github.com/apache/spark/pull/15814#discussion_r87072977"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/15814"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15814","id":92848615,"html_url":"https://github.com/apache/spark/pull/15814","diff_url":"https://github.com/apache/spark/pull/15814.diff","patch_url":"https://github.com/apache/spark/pull/15814.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/15814","number":15814,"state":"open","locked":false,"title":"[SPARK-18185] Support all forms of INSERT / OVERWRITE TABLE for Datasource tables","user":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAs of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table will overwrite the entire table instead of only the partitions matching the static keys, as in Hive. It also doesn't respect custom partition locations.\r\n\r\nThis PR adds support for all these operations to Datasource tables managed by the Hive metastore. It is implemented as follows\r\n- During planning time, the full set of partitions affected by an INSERT or OVERWRITE command is read from the Hive metastore.\r\n- The planner identifies any partitions with custom locations and includes this in the write task metadata.\r\n- FileFormatWriter tasks refer to this custom locations map when determining where to write for dynamic partition output.\r\n- When the write job finishes, the set of written partitions is compared against the initial set of matched partitions, and the Hive metastore is updated to reflect the newly added / removed partitions.\r\n\r\nIt was necessary to introduce a method for staging files with absolute output paths to `FileCommitProtocol`. These files are not handled by the Hadoop output committer but are moved to their final locations when the job commits.\r\n\r\nThe overwrite behavior of legacy Datasource tables is also changed: no longer will the entire table be overwritten if a partial partition spec is present.\r\n\r\ncc @cloud-fan @yhuai \r\n\r\n## How was this patch tested?\r\n\r\nUnit tests, existing tests.","created_at":"2016-11-08T20:02:05Z","updated_at":"2016-11-08T20:09:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1e65f9aca94957896fa93a2e1d56b4a0298606d6","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/15814/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/15814/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/15814/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/b5bbb1d664dd23131635c156b6b8628184e0cc4f","head":{"label":"ericl:sc-5027","ref":"sc-5027","sha":"b5bbb1d664dd23131635c156b6b8628184e0cc4f","user":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"repo":{"id":23644625,"name":"spark","full_name":"ericl/spark","owner":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ericl/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/ericl/spark","forks_url":"https://api.github.com/repos/ericl/spark/forks","keys_url":"https://api.github.com/repos/ericl/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ericl/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ericl/spark/teams","hooks_url":"https://api.github.com/repos/ericl/spark/hooks","issue_events_url":"https://api.github.com/repos/ericl/spark/issues/events{/number}","events_url":"https://api.github.com/repos/ericl/spark/events","assignees_url":"https://api.github.com/repos/ericl/spark/assignees{/user}","branches_url":"https://api.github.com/repos/ericl/spark/branches{/branch}","tags_url":"https://api.github.com/repos/ericl/spark/tags","blobs_url":"https://api.github.com/repos/ericl/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ericl/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ericl/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/ericl/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ericl/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/ericl/spark/languages","stargazers_url":"https://api.github.com/repos/ericl/spark/stargazers","contributors_url":"https://api.github.com/repos/ericl/spark/contributors","subscribers_url":"https://api.github.com/repos/ericl/spark/subscribers","subscription_url":"https://api.github.com/repos/ericl/spark/subscription","commits_url":"https://api.github.com/repos/ericl/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/ericl/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/ericl/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/ericl/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/ericl/spark/contents/{+path}","compare_url":"https://api.github.com/repos/ericl/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ericl/spark/merges","archive_url":"https://api.github.com/repos/ericl/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ericl/spark/downloads","issues_url":"https://api.github.com/repos/ericl/spark/issues{/number}","pulls_url":"https://api.github.com/repos/ericl/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/ericl/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/ericl/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ericl/spark/labels{/name}","releases_url":"https://api.github.com/repos/ericl/spark/releases{/id}","deployments_url":"https://api.github.com/repos/ericl/spark/deployments","created_at":"2014-09-04T01:48:06Z","updated_at":"2016-07-30T22:03:00Z","pushed_at":"2016-11-08T19:50:54Z","git_url":"git://github.com/ericl/spark.git","ssh_url":"git@github.com:ericl/spark.git","clone_url":"https://github.com/ericl/spark.git","svn_url":"https://github.com/ericl/spark","homepage":null,"size":242826,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"245e5a2f80e3195b7f8a38b480b29bfc23af66bf","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-11-08T18:42:42Z","pushed_at":"2016-11-08T20:02:05Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":607587,"stargazers_count":10657,"watchers_count":10657,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9956,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":441,"forks":9956,"open_issues":441,"watchers":10657,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/15814"},"html":{"href":"https://github.com/apache/spark/pull/15814"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/15814"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/15814/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/15814/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/15814/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/b5bbb1d664dd23131635c156b6b8628184e0cc4f"}}}}
{"actor":{"display_login":"ericl"}}