{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14803","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14803/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14803/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14803/events","html_url":"https://github.com/apache/spark/pull/14803","id":173131557,"number":14803,"title":"[SPARK-17153][SQL] Should read partition data when reading new files in filestream without globbing","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-25T07:25:42Z","updated_at":"2016-08-30T16:33:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14803","html_url":"https://github.com/apache/spark/pull/14803","diff_url":"https://github.com/apache/spark/pull/14803.diff","patch_url":"https://github.com/apache/spark/pull/14803.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen reading file stream with non-globbing path, the results return data with all `null`s for the \r\npartitioned columns. E.g.,\r\n\r\n    case class A(id: Int, value: Int)\r\n    val data = spark.createDataset(Seq(\r\n      A(1, 1), \r\n      A(2, 2), \r\n      A(2, 3))\r\n    ) \r\n    val url = \"/tmp/test\"\r\n    data.write.partitionBy(\"id\").parquet(url)\r\n    spark.read.parquet(url).show\r\n\r\n    +-----+---+\r\n    |value| id|\r\n    +-----+---+\r\n    |    2|  2|\r\n    |    3|  2|\r\n    |    1|  1|\r\n    +-----+---+\r\n\r\n    val s = spark.readStream.schema(spark.read.load(url).schema).parquet(url)\r\n    s.writeStream.queryName(\"test\").format(\"memory\").start()\r\n\r\n    sql(\"SELECT * FROM test\").show\r\n\r\n    +-----+----+\r\n    |value|  id|\r\n    +-----+----+\r\n    |    2|null|\r\n    |    3|null|\r\n    |    1|null|\r\n    +-----+----+\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/243499211","html_url":"https://github.com/apache/spark/pull/14803#issuecomment-243499211","issue_url":"https://api.github.com/repos/apache/spark/issues/14803","id":243499211,"user":{"login":"frreiss","id":12436991,"avatar_url":"https://avatars.githubusercontent.com/u/12436991?v=3","gravatar_id":"","url":"https://api.github.com/users/frreiss","html_url":"https://github.com/frreiss","followers_url":"https://api.github.com/users/frreiss/followers","following_url":"https://api.github.com/users/frreiss/following{/other_user}","gists_url":"https://api.github.com/users/frreiss/gists{/gist_id}","starred_url":"https://api.github.com/users/frreiss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frreiss/subscriptions","organizations_url":"https://api.github.com/users/frreiss/orgs","repos_url":"https://api.github.com/users/frreiss/repos","events_url":"https://api.github.com/users/frreiss/events{/privacy}","received_events_url":"https://api.github.com/users/frreiss/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T16:33:01Z","updated_at":"2016-08-30T16:33:01Z","body":"LGTM"}}
{"actor":{"display_login":"frreiss"}}