{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20579","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20579/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20579/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20579/events","html_url":"https://github.com/apache/spark/pull/20579","id":296234399,"number":20579,"title":"[SPARK-23372][SQL] Writing empty struct in parquet fails during execution. It should fail earlier in the processing.","user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars0.githubusercontent.com/u/14225158?v=4","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-02-11T22:50:59Z","updated_at":"2018-03-19T20:18:31Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20579","html_url":"https://github.com/apache/spark/pull/20579","diff_url":"https://github.com/apache/spark/pull/20579.diff","patch_url":"https://github.com/apache/spark/pull/20579.patch"},"body":"## What changes were proposed in this pull request?\r\n**Running**\r\nspark.emptyDataFrame.write.format(\"parquet\").mode(\"overwrite\").save(path)\r\n**Results in**\r\n``` SQL\r\norg.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: message spark_schema {\r\n }\r\n\r\nat org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:27)\r\n at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:37)\r\n at org.apache.parquet.schema.MessageType.accept(MessageType.java:58)\r\n at org.apache.parquet.schema.TypeUtil.checkValidWriteSchema(TypeUtil.java:23)\r\n at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:225)\r\n at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:342)\r\n at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:302)\r\n at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)\r\n at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.newOutputWriter(FileFormatWriter.scala:376)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:387)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:278)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:276)\r\n at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:281)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:206)\r\n at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:205)\r\n at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n at org.apache.spark.scheduler.Task.run(Task.scala:109)\r\n at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)\r\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n at java.lang.Thread.run(Thread.\r\n```\r\n\r\nThis PR addresses the above issue by raising the error earlier.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests added in FileBasedDatasourceSuite."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/374358346","html_url":"https://github.com/apache/spark/pull/20579#issuecomment-374358346","issue_url":"https://api.github.com/repos/apache/spark/issues/20579","id":374358346,"user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars1.githubusercontent.com/u/11567269?v=4","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"created_at":"2018-03-19T20:18:31Z","updated_at":"2018-03-19T20:18:31Z","author_association":"MEMBER","body":"Check whether the format is file-based data sources? Then, we do not need to check the same thing for different file sources."}}
{"actor":{"display_login":"gatorsmile"}}