{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14124","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14124/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14124/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14124/events","html_url":"https://github.com/apache/spark/pull/14124","id":164705353,"number":14124,"title":"[SPARK-16472][SQL] Inconsistent nullability in schema after being read in SQL API","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-07-10T08:17:41Z","updated_at":"2016-07-10T13:44:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14124","html_url":"https://github.com/apache/spark/pull/14124","diff_url":"https://github.com/apache/spark/pull/14124.diff","patch_url":"https://github.com/apache/spark/pull/14124.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIt seems the data sources implementing `FileFormat` seems loading the data by forcing the fields as nullable fields. (See here, https://github.com/apache/spark/blob/7374e518e2641fddfe57003340db410224b37581/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L402-L409)\r\n\r\nIt seems this was officially documented (for Parquet) in SPARK-11360 and was discussed here https://www.mail-archive.com/user@spark.apache.org/msg39230.html\r\n\r\nHowever, I realised that several APIs do not follow this. For example,\r\n\r\n```scala\r\njson(jsonRDD: RDD[String]) // in DataFrameReader\r\n```\r\n\r\nSo, the codes below:\r\n\r\n```scala\r\nval rdd = spark.sparkContext.makeRDD(Seq(\"{\\\"a\\\" : 1}\", \"{\\\"a\\\" : null}\"))\r\nval schema = StructType(StructField(\"a\", IntegerType, nullable = false) :: Nil)\r\nval df = spark.read.schema(schema).json(rdd)\r\ndf.printSchema()\r\n```\r\n\r\nprints below:\r\n\r\n```\r\nroot\r\n |-- a: integer (nullable = false)\r\n```\r\n\r\nThis API loads the schema as it is after loading. However, the schema becomes different when loading it by the API below (nullable fields) :\r\n\r\n```scala\r\nspark.read.format(\"json\").schema(...).load(path).printSchema()\r\n```\r\n\r\n```scala\r\nspark.read.schema(...).json(path).printSchema()\r\n```\r\n\r\nproduce below:\r\n\r\n```\r\nroot\r\n |-- a: integer (nullable = true)\r\n```\r\n\r\nIn addition, this is happening for structured streaming as well. (even when we read batch after writing it by structured streaming).\r\n\r\nI wrote some more such cases in the tests (for Structured Streaming). Please refer the tests codes below.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests in `JsonSuite`, `FileStreamSinkSuite`, `FileStreamSourceSuite` and `HadoopFsRelationTest`.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/231589732","html_url":"https://github.com/apache/spark/pull/14124#issuecomment-231589732","issue_url":"https://api.github.com/repos/apache/spark/issues/14124","id":231589732,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-07-10T13:44:30Z","updated_at":"2016-07-10T13:44:30Z","body":"**[Test build #62054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62054/consoleFull)** for PR 14124 at commit [`3980681`](https://github.com/apache/spark/commit/39806815fbbef2aafb32d3173c23386fcfbc5edf).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}