{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13778","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13778/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13778/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13778/events","html_url":"https://github.com/apache/spark/pull/13778","id":161128386,"number":13778,"title":"[SPARK-16062][SPARK-15989][SQL] Fix two bugs of Python-only UDTs","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-06-20T06:57:15Z","updated_at":"2016-07-08T03:16:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13778","html_url":"https://github.com/apache/spark/pull/13778","diff_url":"https://github.com/apache/spark/pull/13778.diff","patch_url":"https://github.com/apache/spark/pull/13778.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThere are two related bugs of Python-only UDTs. Because the test case of second one needs the first fix too. I put them into one PR. If it is not appropriate, please let me know.\r\n\r\n### First bug: When MapObjects works on Python-only UDTs\r\n\r\n`RowEncoder` will use `PythonUserDefinedType.sqlType` for its deserializer expression. If the sql type is `ArrayType`, we will have `MapObjects` working on it. But `MapObjects` doesn't consider `PythonUserDefinedType` as its input data type. It causes error like:\r\n\r\n    import pyspark.sql.group\r\n    from pyspark.sql.tests import PythonOnlyPoint, PythonOnlyUDT\r\n    from pyspark.sql.types import *\r\n\r\n    schema = StructType().add(\"key\", LongType()).add(\"val\", PythonOnlyUDT())\r\n    df = spark.createDataFrame([(i % 3, PythonOnlyPoint(float(i), float(i))) for i in range(10)], schema=schema)\r\n    df.show()\r\n\r\n    File \"/home/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py\", line 312, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o36.showString.\r\n    : java.lang.RuntimeException: Error while decoding: scala.MatchError: org.apache.spark.sql.types.PythonUserDefinedType@f4ceede8 (of class org.apache.spark.sql.types.PythonUserDefinedType)\r\n    ...\r\n\r\n### Second bug: When Python-only UDTs is the element type of ArrayType\r\n\r\n    import pyspark.sql.group\r\n    from pyspark.sql.tests import PythonOnlyPoint, PythonOnlyUDT\r\n    from pyspark.sql.types import *\r\n\r\n    schema = StructType().add(\"key\", LongType()).add(\"val\", ArrayType(PythonOnlyUDT()))\r\n    df = spark.createDataFrame([(i % 3, [PythonOnlyPoint(float(i), float(i))]) for i in range(10)], schema=schema)\r\n    df.show()\r\n\r\n## How was this patch tested?\r\nPySpark's sql tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/231265743","html_url":"https://github.com/apache/spark/pull/13778#issuecomment-231265743","issue_url":"https://api.github.com/repos/apache/spark/issues/13778","id":231265743,"user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2016-07-08T03:16:46Z","updated_at":"2016-07-08T03:16:46Z","body":"Python UDT in python side only serializes the python data to sql type defined in the Python UDT. The problem now is happened at the serialization to row in java side on the serialized python data. I think it can not be certain that the serialized python data doesn't need the serialization in java side."}}
{"actor":{"display_login":"viirya"}}