{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13486","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13486/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13486/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13486/events","html_url":"https://github.com/apache/spark/pull/13486","id":158267320,"number":13486,"title":"[SPARK-15743][SQL] Prevent saving with all-column partitioning","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-06-03T00:18:33Z","updated_at":"2016-06-04T10:16:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13486","html_url":"https://github.com/apache/spark/pull/13486","diff_url":"https://github.com/apache/spark/pull/13486.diff","patch_url":"https://github.com/apache/spark/pull/13486.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen saving datasets on storage, `partitionBy` provides an easy way to construct the directory structure. However, if a user choose all columns as partition columns, some exceptions occurs.\r\n\r\n- **ORC with all column partitioning**: `AnalysisException` on **future read** due to schema inference failure.\r\n ```scala\r\nscala> spark.range(10).write.format(\"orc\").mode(\"overwrite\").partitionBy(\"id\").save(\"/tmp/data\")\r\n                                                                                \r\nscala> spark.read.format(\"orc\").load(\"/tmp/data\").collect()\r\norg.apache.spark.sql.AnalysisException: Unable to infer schema for ORC at /tmp/data. It must be specified manually;\r\n```\r\n\r\n- **Parquet with all-column partitioning**: `InvalidSchemaException` on **write execution** due to Parquet limitation.\r\n ```scala\r\nscala> spark.range(100).write.format(\"parquet\").mode(\"overwrite\").partitionBy(\"id\").save(\"/tmp/data\")\r\n[Stage 0:>                                                          (0 + 8) / 8]16/06/02 16:51:17\r\nERROR Utils: Aborting task\r\norg.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema\r\n... (lots of error messages)\r\n```\r\n\r\nAlthough some formats like JSON support all-column partitioning without any problem, it seems not a good idea to make lots of empty directories. \r\n\r\nThis PR prevents saving with all-column partitioning by consistently raising `AnalysisException` before executing save operation. \r\n\r\n## How was this patch tested?\r\n\r\nNewly added `PartitioningUtilsSuite`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/223748030","html_url":"https://github.com/apache/spark/pull/13486#issuecomment-223748030","issue_url":"https://api.github.com/repos/apache/spark/issues/13486","id":223748030,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-06-04T10:16:06Z","updated_at":"2016-06-04T10:16:06Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59986/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}