{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/55117300","id":55117300,"diff_hunk":"@@ -278,26 +298,61 @@ object ResolvedDataSource extends Logging {\n         val equality = columnNameEquality(caseSensitive)\n         val dataSchema = StructType(\n           data.schema.filterNot(f => partitionColumns.exists(equality(_, f.name))))\n-        val r = dataSource.createRelation(\n-          sqlContext,\n-          Array(outputPath.toString),\n-          Some(dataSchema.asNullable),\n-          Some(partitionColumnsSchema(data.schema, partitionColumns, caseSensitive)),\n-          bucketSpec,\n-          caseInsensitiveOptions)\n+\n+        // If we are appending to a table that already exists, make sure the partitioning matches\n+        // up.  If we fail to load the table for whatever reason, ignore the check.\n+        if (mode == SaveMode.Append) {\n+          val existingPartitionColumnSet = try {\n+            val resolved = apply(\n+              sqlContext,\n+              userSpecifiedSchema = Some(data.schema.asNullable),\n+              provider = provider,\n+              options = options)\n+\n+            Some(resolved.relation\n+              .asInstanceOf[HadoopFsRelation]\n+              .location\n+              .partitionSpec(None)\n+              .partitionColumns\n+              .fieldNames\n+              .toSet)\n+          } catch {\n+            case e: Exception =>\n+              None\n+          }\n+\n+          existingPartitionColumnSet.foreach { ex =>\n+            if (ex.map(_.toLowerCase) != partitionColumns.map(_.toLowerCase()).toSet) {\n+              throw new AnalysisException(s\"$ex ${partitionColumns.toSet}\")","path":"sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala","position":296,"original_position":296,"commit_id":"4b53adb1ee5675a56d974c91835918d4b1b4f4f4","original_commit_id":"4b53adb1ee5675a56d974c91835918d4b1b4f4f4","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"body":"Could you make this error message more readable?","created_at":"2016-03-05T07:34:45Z","updated_at":"2016-03-05T07:34:45Z","html_url":"https://github.com/apache/spark/pull/11509#discussion_r55117300","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/11509","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/55117300"},"html":{"href":"https://github.com/apache/spark/pull/11509#discussion_r55117300"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/11509"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11509","id":61661485,"html_url":"https://github.com/apache/spark/pull/11509","diff_url":"https://github.com/apache/spark/pull/11509.diff","patch_url":"https://github.com/apache/spark/pull/11509.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/11509","number":11509,"state":"open","locked":false,"title":"[SPARK-13665][SQL] Separate the concerns of HadoopFsRelation","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"body":"`HadoopFsRelation` is used for reading most files into Spark SQL.  However today this class mixes the concerns of file management, schema reconciliation, scan building, bucketing, partitioning, and writing data.  As a result, many data sources are forced to reimplement the same functionality and the various layers have accumulated a fair bit of inefficiency.  This PR is a first cut at separating this into several components / interfaces that are each described below.  Additionally, all implementations inside of Spark (parquet, csv, json, text, orc, svmlib) have been ported to the new API `FileFormat`.  External libraries, such as spark-avro will also need to be ported to work with Spark 2.0.\r\n\r\n### HadoopFsRelation\r\nA simple `case class` that acts as a container for all of the metadata required to read from a datasource.  All discovery, resolution and merging logic for schemas and partitions has been removed.  This an internal representation that no longer needs to be exposed to developers.\r\n\r\n```scala\r\ncase class HadoopFsRelation(\r\n    sqlContext: SQLContext,\r\n    location: FileCatalog,\r\n    partitionSchema: StructType,\r\n    dataSchema: StructType,\r\n    bucketSpec: Option[BucketSpec],\r\n    fileFormat: FileFormat,\r\n    options: Map[String, String]) extends BaseRelation\r\n```\r\n\r\n### FileFormat\r\nThe primary interface that will be implemented by each different format including external libraries.  Implementors are responsible for reading a given format and converting it into `InternalRow` as well as writing out an `InternalRow`.  A format can optionally return a schema that is inferred from a set of files.\r\n\r\n```scala\r\ntrait FileFormat {\r\n  def inferSchema(\r\n      sqlContext: SQLContext,\r\n      options: Map[String, String],\r\n      files: Seq[FileStatus]): Option[StructType]\r\n\r\n  def prepareWrite(\r\n      sqlContext: SQLContext,\r\n      job: Job,\r\n      options: Map[String, String],\r\n      dataSchema: StructType): OutputWriterFactory\r\n\r\n  def buildInternalScan(\r\n      sqlContext: SQLContext,\r\n      dataSchema: StructType,\r\n      requiredColumns: Array[String],\r\n      filters: Array[Filter],\r\n      bucketSet: Option[BitSet],\r\n      inputFiles: Array[FileStatus],\r\n      broadcastedConf: Broadcast[SerializableConfiguration],\r\n      options: Map[String, String]): RDD[InternalRow]\r\n}\r\n```\r\n\r\nThe current interface is based on what was required to get all the tests passing again, but still mixes a couple of concerns (i.e. `bucketSet` is passed down to the scan instead of being resolved by the planner).  Additionally, scans are still returning `RDD`s instead of iterators for single files.  In a future PR, bucketing should be removed from this interface and the scan should be isolated to a single file.\r\n\r\n### FileCatalog\r\nThis interface is used to list the files that make up a given relation, as well as handle directory based partitioning.\r\n\r\n```scala\r\ntrait FileCatalog {\r\n  def paths: Seq[Path]\r\n  def partitionSpec(schema: Option[StructType]): PartitionSpec\r\n  def allFiles(): Seq[FileStatus]\r\n  def getStatus(path: Path): Array[FileStatus]\r\n  def refresh(): Unit\r\n}\r\n```\r\n\r\nCurrently there are two implementations:\r\n - `HDFSFileCatalog` - based on code from the old `HadoopFsRelation`.  Infers partitioning by recursive listing and caches this data for performance\r\n - `HiveFileCatalog` - based on the above, but it uses the partition spec from the Hive Metastore.\r\n\r\n### ResolvedDataSource\r\nProduces a logical plan given the following description of a Data Source (which can come from DataFrameReader or a metastore): \r\n - `paths: Seq[String] = Nil`\r\n - `userSpecifiedSchema: Option[StructType] = None`\r\n - `partitionColumns: Array[String] = Array.empty`\r\n - `bucketSpec: Option[BucketSpec] = None`\r\n - `provider: String`\r\n - `options: Map[String, String]`\r\n\r\nThis class is responsible for deciding which of the Data Source APIs a given provider is using (including the non-file based ones).  All reconciliation of partitions, buckets, schema from metastores or inference is done here.\r\n\r\n### DataSourceAnalysis / DataSourceStrategy\r\nResponsible for analyzing and planning reading/writing of data using any of the Data Source APIs, including:\r\n - pruning the files from partitions that will be read based on filters.\r\n - appending partition columns*\r\n - applying additional filters when a data source can not evaluate them internally.\r\n - constructing an RDD that is bucketed correctly when required*\r\n - sanity checking schema match-up and other analysis when writing.\r\n\r\n*In the future we should do that following:\r\n - Break out file handling into its own Strategy as its sufficiently complex / isolated.\r\n - Push the appending of partition columns down in to `FileFormat` to avoid an extra copy / unvectorization.\r\n - Use a custom RDD for scans instead of `SQLNewNewHadoopRDD2`\r\n\r\nTODO:\r\n - [ ] Better initial scala doc on the interfaces\r\n - [ ] Create follow-up JIRAs for the TODOs described above and in the code.","created_at":"2016-03-04T03:10:26Z","updated_at":"2016-03-05T07:34:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"52af9d9089852443681c8d3bd7b2e18ef22661e3","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/11509/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/11509/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/11509/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/4b53adb1ee5675a56d974c91835918d4b1b4f4f4","head":{"label":"marmbrus:fileDataSource","ref":"fileDataSource","sha":"4b53adb1ee5675a56d974c91835918d4b1b4f4f4","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"repo":{"id":17230869,"name":"spark","full_name":"marmbrus/spark","owner":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/marmbrus/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/marmbrus/spark","forks_url":"https://api.github.com/repos/marmbrus/spark/forks","keys_url":"https://api.github.com/repos/marmbrus/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/marmbrus/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/marmbrus/spark/teams","hooks_url":"https://api.github.com/repos/marmbrus/spark/hooks","issue_events_url":"https://api.github.com/repos/marmbrus/spark/issues/events{/number}","events_url":"https://api.github.com/repos/marmbrus/spark/events","assignees_url":"https://api.github.com/repos/marmbrus/spark/assignees{/user}","branches_url":"https://api.github.com/repos/marmbrus/spark/branches{/branch}","tags_url":"https://api.github.com/repos/marmbrus/spark/tags","blobs_url":"https://api.github.com/repos/marmbrus/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/marmbrus/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/marmbrus/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/marmbrus/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/marmbrus/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/marmbrus/spark/languages","stargazers_url":"https://api.github.com/repos/marmbrus/spark/stargazers","contributors_url":"https://api.github.com/repos/marmbrus/spark/contributors","subscribers_url":"https://api.github.com/repos/marmbrus/spark/subscribers","subscription_url":"https://api.github.com/repos/marmbrus/spark/subscription","commits_url":"https://api.github.com/repos/marmbrus/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/marmbrus/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/marmbrus/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/marmbrus/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/marmbrus/spark/contents/{+path}","compare_url":"https://api.github.com/repos/marmbrus/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/marmbrus/spark/merges","archive_url":"https://api.github.com/repos/marmbrus/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/marmbrus/spark/downloads","issues_url":"https://api.github.com/repos/marmbrus/spark/issues{/number}","pulls_url":"https://api.github.com/repos/marmbrus/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/marmbrus/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/marmbrus/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/marmbrus/spark/labels{/name}","releases_url":"https://api.github.com/repos/marmbrus/spark/releases{/id}","deployments_url":"https://api.github.com/repos/marmbrus/spark/deployments","created_at":"2014-02-27T00:22:54Z","updated_at":"2014-11-27T06:21:59Z","pushed_at":"2016-03-05T00:04:57Z","git_url":"git://github.com/marmbrus/spark.git","ssh_url":"git@github.com:marmbrus/spark.git","clone_url":"https://github.com/marmbrus/spark.git","svn_url":"https://github.com/marmbrus/spark","homepage":null,"size":193986,"stargazers_count":2,"watchers_count":2,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":1,"forks":2,"open_issues":1,"watchers":2,"default_branch":"catalyst"}},"base":{"label":"apache:master","ref":"master","sha":"a6e2bd31f52f9e9452e52ab5b846de3dee8b98a7","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-03-05T06:23:55Z","pushed_at":"2016-03-05T07:07:13Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":442097,"stargazers_count":7680,"watchers_count":7680,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":6999,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":394,"forks":6999,"open_issues":394,"watchers":7680,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/11509"},"html":{"href":"https://github.com/apache/spark/pull/11509"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/11509"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/11509/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/11509/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/11509/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/4b53adb1ee5675a56d974c91835918d4b1b4f4f4"}}}}