{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16030","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16030/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16030/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16030/events","html_url":"https://github.com/apache/spark/pull/16030","id":191942004,"number":16030,"title":"[SPARK-18108][SQL] Fix a bug to fail partition schema inference","user":{"login":"maropu","id":692303,"avatar_url":"https://avatars.githubusercontent.com/u/692303?v=3","gravatar_id":"","url":"https://api.github.com/users/maropu","html_url":"https://github.com/maropu","followers_url":"https://api.github.com/users/maropu/followers","following_url":"https://api.github.com/users/maropu/following{/other_user}","gists_url":"https://api.github.com/users/maropu/gists{/gist_id}","starred_url":"https://api.github.com/users/maropu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maropu/subscriptions","organizations_url":"https://api.github.com/users/maropu/orgs","repos_url":"https://api.github.com/users/maropu/repos","events_url":"https://api.github.com/users/maropu/events{/privacy}","received_events_url":"https://api.github.com/users/maropu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-11-28T08:08:30Z","updated_at":"2016-12-02T10:37:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16030","html_url":"https://github.com/apache/spark/pull/16030","diff_url":"https://github.com/apache/spark/pull/16030.diff","patch_url":"https://github.com/apache/spark/pull/16030.patch"},"body":"## What changes were proposed in this pull request?\r\nThis pr is to fix a bug to fail partition schema inference;\r\n```\r\nscala> case class A(a: Long, b: Int)\r\nscala> val as = Seq(A(1, 2))\r\nscala> spark.createDataFrame(as).write.parquet(\"/data/a=1/\")\r\nscala> val df = spark.read.parquet(\"/data/\")\r\nscala> df.printSchema\r\nroot\r\n |-- a: long (nullable = true)\r\n |-- b: integer (nullable = true)\r\nscala> df.collect\r\njava.lang.NullPointerException\r\n        at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.getLong(OnHeapColumnVector.java:283)\r\n        at org.apache.spark.sql.execution.vectorized.ColumnarBatch$Row.getLong(ColumnarBatch.java:191)\r\n        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)\r\n        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)\r\n        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\r\n        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\r\n```\r\n\r\nThis was because spark failed to infer the partition column as `LongType` and it wrongly regarded the column as `IntegerType` in `DataSource`. Therefore, the query failed in scanning the column from a parquet file. This pr modifies code to throw an exception if data columns are used as partition ones in path.\r\n\r\n## How was this patch tested?\r\nAdd tests in `ParquetPartitionDiscoverySuite`"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264424714","html_url":"https://github.com/apache/spark/pull/16030#issuecomment-264424714","issue_url":"https://api.github.com/repos/apache/spark/issues/16030","id":264424714,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T10:37:34Z","updated_at":"2016-12-02T10:37:34Z","body":"**[Test build #69565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69565/consoleFull)** for PR 16030 at commit [`34a324a`](https://github.com/apache/spark/commit/34a324a74f1375cd8871e5cf2aac1318529b944f)."}}
{"actor":{"display_login":"SparkQA"}}