{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15361","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15361/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15361/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15361/events","html_url":"https://github.com/apache/spark/pull/15361","id":181120455,"number":15361,"title":"[SPARK-17765][SQL] Support for writing out user-defined type in ORC datasource","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-05T10:44:55Z","updated_at":"2016-10-05T12:16:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15361","html_url":"https://github.com/apache/spark/pull/15361","diff_url":"https://github.com/apache/spark/pull/15361.diff","patch_url":"https://github.com/apache/spark/pull/15361.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR adds the support for `UserDefinedType` when writing out instead of throwing `ClassCastException` in ORC data source.\r\n\r\nIn more details, `OrcStruct` is being created based on string from`DataType.catalogString`. For user-defined type, it seems it returns `sqlType.simpleString` for `catalogString` by default[1]. However, during type-dispatching to match the output with the schema, it tries to cast to, for example, `StructType`[2]. \r\n\r\nSo, running the codes below (`MyDenseVector` was borrowed[3]) :\r\n\r\n```scala\r\nval data = Seq((1, new UDT.MyDenseVector(Array(0.25, 2.25, 4.25))))\r\nval udtDF = data.toDF(\"id\", \"vectors\")\r\nudtDF.write.orc(\"/tmp/test.orc\")\r\n```\r\n\r\nends up throwing an exception as below:\r\n\r\n```\r\njava.lang.ClassCastException: org.apache.spark.sql.UDT$MyDenseVectorUDT cannot be cast to org.apache.spark.sql.types.ArrayType\r\n\tat org.apache.spark.sql.hive.HiveInspectors$class.wrapperFor(HiveInspectors.scala:381)\r\n\tat org.apache.spark.sql.hive.orc.OrcSerializer.wrapperFor(OrcFileFormat.scala:164)\r\n...\r\n```\r\n\r\nSo, this PR uses `UserDefinedType.sqlType` during finding the correct converter when writing out in ORC data source.\r\n\r\n[1]https://github.com/apache/spark/blob/dfdcab00c7b6200c22883baa3ebc5818be09556f/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UserDefinedType.scala#L95\r\n[2]https://github.com/apache/spark/blob/d2dc8c4a162834818190ffd82894522c524ca3e5/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala#L326\r\n[3]https://github.com/apache/spark/blob/2bfed1a0c5be7d0718fd574a4dad90f4f6b44be7/sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala#L38-L70\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests in `OrcQuerySuite`.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/251657778","html_url":"https://github.com/apache/spark/pull/15361#issuecomment-251657778","issue_url":"https://api.github.com/repos/apache/spark/issues/15361","id":251657778,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-05T12:16:28Z","updated_at":"2016-10-05T12:16:28Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66386/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}