{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13873","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13873/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13873/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13873/events","html_url":"https://github.com/apache/spark/pull/13873","id":161870659,"number":13873,"title":"[SPARK-16167][SQL] RowEncoder should preserve array/map type nullability.","user":{"login":"ueshin","id":506656,"avatar_url":"https://avatars.githubusercontent.com/u/506656?v=3","gravatar_id":"","url":"https://api.github.com/users/ueshin","html_url":"https://github.com/ueshin","followers_url":"https://api.github.com/users/ueshin/followers","following_url":"https://api.github.com/users/ueshin/following{/other_user}","gists_url":"https://api.github.com/users/ueshin/gists{/gist_id}","starred_url":"https://api.github.com/users/ueshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ueshin/subscriptions","organizations_url":"https://api.github.com/users/ueshin/orgs","repos_url":"https://api.github.com/users/ueshin/repos","events_url":"https://api.github.com/users/ueshin/events{/privacy}","received_events_url":"https://api.github.com/users/ueshin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-06-23T08:48:44Z","updated_at":"2016-07-06T08:57:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13873","html_url":"https://github.com/apache/spark/pull/13873","diff_url":"https://github.com/apache/spark/pull/13873.diff","patch_url":"https://github.com/apache/spark/pull/13873.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently `RowEncoder` doesn't preserve nullability of `ArrayType` or `MapType`.\r\nIt returns always `containsNull = true` for `ArrayType`, `valueContainsNull = true` for `MapType` and also the nullability of itself is always `true`.\r\n\r\nThis pr fixes the nullability of them.\r\n\r\n## How was this patch tested?\r\n\r\nAdd tests to check if `RowEncoder` preserves array/map nullability.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/230716374","html_url":"https://github.com/apache/spark/pull/13873#issuecomment-230716374","issue_url":"https://api.github.com/repos/apache/spark/issues/13873","id":230716374,"user":{"login":"ueshin","id":506656,"avatar_url":"https://avatars.githubusercontent.com/u/506656?v=3","gravatar_id":"","url":"https://api.github.com/users/ueshin","html_url":"https://github.com/ueshin","followers_url":"https://api.github.com/users/ueshin/followers","following_url":"https://api.github.com/users/ueshin/following{/other_user}","gists_url":"https://api.github.com/users/ueshin/gists{/gist_id}","starred_url":"https://api.github.com/users/ueshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ueshin/subscriptions","organizations_url":"https://api.github.com/users/ueshin/orgs","repos_url":"https://api.github.com/users/ueshin/repos","events_url":"https://api.github.com/users/ueshin/events{/privacy}","received_events_url":"https://api.github.com/users/ueshin/received_events","type":"User","site_admin":false},"created_at":"2016-07-06T08:57:29Z","updated_at":"2016-07-06T08:57:29Z","body":"@cloud-fan Thank you for your review.\r\nI thought the problem is the schema would change after serialized by the `RowEncoder` because the schema after serialization will be retrieved from the serializer's attributes.\r\n\r\nFor example, the following test will fail:\r\n\r\n```\r\n    val schema = new StructType()\r\n      .add(\"array\", ArrayType(StringType, containsNull = false), nullable = false)\r\n    val ds1 = ...\r\n    val ds2 = ds1.map(_ => Row(Array(\"1\", \"2\", \"3\")))(RowEncoder(schema))\r\n    assert(ds2.schema === schema)\r\n```\r\n\r\nbecause `ds2.schema` becomes before this patch:\r\n\r\n```\r\nroot\r\n |-- array: array (nullable = true)\r\n |    |-- element: string (containsNull = true)\r\n```\r\n\r\n, whereas after this patch (and this would be desirable):\r\n\r\n```\r\nroot\r\n |-- array: array (nullable = false)\r\n |    |-- element: string (containsNull = false)\r\n```\r\n"}}
{"actor":{"display_login":"ueshin"}}