{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21732","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21732/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21732/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21732/events","html_url":"https://github.com/apache/spark/pull/21732","id":339295451,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk5OTg0ODEx","number":21732,"title":"[SPARK-24762][SQL] Enable Option of Product encoders","user":{"login":"viirya","id":68855,"node_id":"MDQ6VXNlcjY4ODU1","avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":93,"created_at":"2018-07-09T03:43:53Z","updated_at":"2018-12-27T14:51:30Z","closed_at":"2018-11-26T03:17:51Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21732","html_url":"https://github.com/apache/spark/pull/21732","diff_url":"https://github.com/apache/spark/pull/21732.diff","patch_url":"https://github.com/apache/spark/pull/21732.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSparkSQL doesn't support to encode `Option[Product]` as a top-level row now, because in SparkSQL entire top-level row can't be null.\r\n\r\nHowever for use cases like Aggregator, it is reasonable to use `Option[Product]` as buffer and output column types. Due to above limitation, we don't do it for now.\r\n\r\nThis patch proposes to encode `Option[Product]` at top-level as single struct column. So we can work around the issue that entire top-level row can't be null.\r\n\r\nTo summarize encoding of `Product` and `Option[Product]`.\r\n\r\nFor `Product`, 1. at root level, the schema is all fields are flatten it into multiple columns. The `Product ` can't be null, otherwise it throws an exception.\r\n\r\n```scala\r\nval df = Seq((1 -> \"a\"), (2 -> \"b\")).toDF()\r\ndf.printSchema()\r\n\r\nroot\r\n |-- _1: integer (nullable = false)\r\n |-- _2: string (nullable = true)\r\n```\r\n\r\n2. At non-root level, `Product` is a struct type column.\r\n\r\n```scala\r\nval df = Seq((1, (1 -> \"a\")), (2, (2 -> \"b\")), (3, null)).toDF()\r\ndf.printSchema()\r\n\r\nroot\r\n |-- _1: integer (nullable = false)\r\n |-- _2: struct (nullable = true)\r\n |    |-- _1: integer (nullable = false)\r\n |    |-- _2: string (nullable = true)\r\n```\r\n\r\nFor `Option[Product]`, 1. it was not supported at root level. After this change, it is a struct type column.\r\n\r\n```scala\r\nval df = Seq(Some(1 -> \"a\"), Some(2 -> \"b\"), None).toDF()\r\ndf.printSchema\r\n\r\nroot\r\n |-- value: struct (nullable = true)\r\n |    |-- _1: integer (nullable = false)\r\n |    |-- _2: string (nullable = true)\r\n```\r\n\r\n2. At non-root level, it is also a struct type column.\r\n\r\n```scala\r\nval df = Seq((1, Some(1 -> \"a\")), (2, Some(2 -> \"b\")), (3, None)).toDF()\r\ndf.printSchema\r\n\r\nroot\r\n |-- _1: integer (nullable = false)\r\n |-- _2: struct (nullable = true)\r\n |    |-- _1: integer (nullable = false)\r\n |    |-- _2: string (nullable = true)\r\n```\r\n\r\n3. For use case like Aggregator, it was not supported too. After this change, we support to use `Option[Product]` as buffer/output column type.\r\n\r\n```scala\r\nval df = Seq(\r\n    OptionBooleanIntData(\"bob\", Some((true, 1))),\r\n    OptionBooleanIntData(\"bob\", Some((false, 2))),\r\n    OptionBooleanIntData(\"bob\", None)).toDF()\r\n\r\nval group = df\r\n    .groupBy(\"name\")\r\n    .agg(OptionBooleanIntAggregator(\"isGood\").toColumn.alias(\"isGood\"))\r\ngroup.printSchema\r\n\r\nroot                                                  \r\n |-- name: string (nullable = true)                          \r\n |-- isGood: struct (nullable = true)\r\n |    |-- _1: boolean (nullable = false)\r\n |    |-- _2: integer (nullable = false)\r\n```\r\n\r\nThe buffer and output type of `OptionBooleanIntAggregator` is both `Option[(Boolean, Int)`.\r\n\r\n## How was this patch tested?\r\n\r\nAdded test.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/450164376","html_url":"https://github.com/apache/spark/pull/21732#issuecomment-450164376","issue_url":"https://api.github.com/repos/apache/spark/issues/21732","id":450164376,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDE2NDM3Ng==","user":{"login":"viirya","id":68855,"node_id":"MDQ6VXNlcjY4ODU1","avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2018-12-27T14:51:30Z","updated_at":"2018-12-27T14:51:30Z","author_association":"CONTRIBUTOR","body":"@HyukjinKwon there was a comment https://github.com/apache/spark/pull/21732#issuecomment-403865941 for it. This was originally to make `Option[Product]` work for aggregators, but it makes encoder framework inconsistent because the created encoders can used for aggregators but not for Dataset. So I think the idea is to make it consistent. Users are allowed to encoder `Option[Product]` as Dataset but they should be aware of the encoding."}}
{"actor":{"display_login":"viirya"}}