{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14045","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14045/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14045/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14045/events","html_url":"https://github.com/apache/spark/pull/14045","id":163640640,"number":14045,"title":"[SPARK-16362][SQL] Support ArrayType and StructType in vectorization Parquet reader","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2016-07-04T09:44:42Z","updated_at":"2016-07-20T08:01:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14045","html_url":"https://github.com/apache/spark/pull/14045","diff_url":"https://github.com/apache/spark/pull/14045.diff","patch_url":"https://github.com/apache/spark/pull/14045.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nVectorization parquet reader now doesn't support complex types such as ArrayType, MapType and StructType. We should support it to extend the coverage of performance improvement introduced by vectorization parquet reader. This patch is to add ArrayType and StructType first.\r\n\r\n### Main changes\r\n\r\n* Obtain repetition and definition level values during converting Parquet schema\r\n\r\n  We convert Parquet schema to Catalyst DataType in `ParquetSchemaConverter`. Because we need repetition and definition level information during constructing complex types back from Parquet data, this PR obtains the repetition and definition levels for complex types and attaches them to Catalyst `ArrayType`, `MapType` and `StructType`. Accordingly, this PR adds a metadata to these Calalyst DataTypes as a new attribute. Although this PR tries to avoid modifying these Catalyst DataTypes, however, because the column vector in vectorization is corresponding to Catalyst DataType instead of Parquet schema, we can't access the repetition and definition levels for each column if we don't attach them to their assigned DataType.\r\n\r\n* Attach `VectorizedColumnReader`  to `ColumnVector`\r\n\r\n  Because in flat schema each `ColumnVector` is actually a data column, previously the relation between `VectorizedColumnReader` and `ColumnVector` is one-by-one. Now only the `ColumnVector` representing a data column will have corresponding `VectorizedColumnReader`. Then when it is time to read batch, the `ColumnVector` with complex type will delegate to its child `ColumnVector`.\r\n\r\n* Implement constructing complex records in `VectorizedColumnReader`\r\n\r\n  The `readBatch` in `VectorizedColumnReader` is the main method to read data into `ColumnVector`. Previously its behavior is simply to load required number of data according to the data type of the column vector. Now after the data is loaded into the column, we need to construct complex records in its parent column that could be an ArrayType, MapType or StructType. The way to restore the data as complex types is encoding in repetition and definition levels in Parquet. The new method `constructComplexRecords` in `VectorizedColumnReader` implements the logic to restore the complex data. Basically, what `constructComplexRecords` does is to count the continuous values and add array into the parent column if the repetition level value indicates a new record happens. Besides, `constructComplexRecords` also needs to consider the null values. Null values could mean a null record at root level, an empty array or struct. This method considers different cases and sets it correctly.\r\n\r\n### Benchmark\r\n\r\n    val N = 10000\r\n    withParquetTable((0 until N).map { i =>\r\n      ((i to i + 1000).toList, (i to i + 100).map(_.toString).toList,\r\n        (i to i + 1000).map(_.toDouble / 2).toList,\r\n        ((0 to 10).map(_.toString).toList, (0 to 10).map(_.toString).toList))\r\n    }, \"t\") {\r\n      val benchmark = new Benchmark(\"Vectorization Parquet for nested types\", N)\r\n      benchmark.addCase(\"Vectorization Parquet reader\", 10) { iter =>\r\n        sql(\"SELECT _1[10], _2[20], _3[30], _4._1[5], _4._2[5] FROM t\").collect()\r\n      }\r\n      benchmark.run()\r\n    }\r\n\r\nDisabled vectorization:\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Vectorization Parquet for nested types:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    Vectorization Parquet reader                  1706 / 2207          0.0      170580.8       1.0X\r\n\r\nEnabled vectorization:\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Vectorization Parquet for nested types:  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    Vectorization Parquet reader                   789 /  972          0.0       78919.4       1.0X\r\n\r\n\r\n## How was this patch tested?\r\n\r\nExisting unit tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/233874573","html_url":"https://github.com/apache/spark/pull/14045#issuecomment-233874573","issue_url":"https://api.github.com/repos/apache/spark/issues/14045","id":233874573,"user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"created_at":"2016-07-20T08:01:36Z","updated_at":"2016-07-20T08:01:36Z","body":"@viirya Thanks for your work! This would be very useful. I'll help review this one soon after finishing my 2.0 tasks at hand!"}}
{"actor":{"display_login":"liancheng"}}