{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13701","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13701/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13701/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13701/events","html_url":"https://github.com/apache/spark/pull/13701","id":160572833,"number":13701,"title":"[SPARK-15639][SPARK-16321][SQL] Push down filter at RowGroups level for parquet reader","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":65,"created_at":"2016-06-16T04:00:08Z","updated_at":"2016-08-05T05:22:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13701","html_url":"https://github.com/apache/spark/pull/13701","diff_url":"https://github.com/apache/spark/pull/13701.diff","patch_url":"https://github.com/apache/spark/pull/13701.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThe base class `SpecificParquetRecordReaderBase` used for vectorized parquet reader will try to get pushed-down filters from the given configuration. This pushed-down filters are used for RowGroups-level filtering. However, we don't set up the filters to push down into the configuration. In other words, the filters are not actually pushed down to do RowGroups-level filtering. This patch is to fix this and tries to set up the filters for pushing down to configuration for the reader.\r\n\r\nThe benchmark that excludes the time of writing Parquet file:\r\n\r\n    test(\"Benchmark for Parquet\") {\r\n      val N = 500 << 12\r\n        withParquetTable((0 until N).map(i => (101, i)), \"t\") {\r\n          val benchmark = new Benchmark(\"Parquet reader\", N)\r\n          benchmark.addCase(\"reading Parquet file\", 10) { iter =>\r\n            sql(\"SELECT _1 FROM t where t._1 < 100\").collect()\r\n          }\r\n          benchmark.run()\r\n      }\r\n    }\r\n\r\n`withParquetTable` in default will run tests for vectorized reader non-vectorized readers. I only let it run vectorized reader.\r\n\r\nWhen we set the block size of parquet as 1024 to have multiple row groups. The benchmark is:\r\n\r\nBefore this patch:\r\n\r\nThe retrieved row groups: 8063\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Parquet reader:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    reading Parquet file                           825 / 1233          2.5         402.6       1.0X\r\n\r\nAfter this patch:\r\n\r\nThe retrieved row groups: 0\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Parquet reader:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    reading Parquet file                           306 /  503          6.7         149.6       1.0X\r\n\r\nNext, I run the benchmark for non-pushdown case using the same benchmark code but with disabled pushdown configuration. This time the parquet block size is default value.\r\n\r\nBefore this patch:\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Parquet reader:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    reading Parquet file                           136 /  238         15.0          66.5       1.0X\r\n\r\nAfter this patch:\r\n\r\n    Java HotSpot(TM) 64-Bit Server VM 1.8.0_71-b15 on Linux 3.19.0-25-generic\r\n    Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz\r\n    Parquet reader:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n    ------------------------------------------------------------------------------------------------\r\n    reading Parquet file                           124 /  193         16.5          60.7       1.0X\r\n\r\nFor non-pushdown case, from the results, I think this patch doesn't affect normal code path.\r\n\r\nI've manually output the `totalRowCount` in `SpecificParquetRecordReaderBase` to see if this patch actually filter the row-groups. When running the above benchmark:\r\n\r\nAfter this patch:\r\n    `totalRowCount = 0`\r\n\r\nBefore this patch:\r\n    `totalRowCount = 1024000`\r\n\r\n\r\n## How was this patch tested?\r\nExisting tests should be passed.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/237758019","html_url":"https://github.com/apache/spark/pull/13701#issuecomment-237758019","issue_url":"https://api.github.com/repos/apache/spark/issues/13701","id":237758019,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-08-05T05:22:42Z","updated_at":"2016-08-05T05:22:42Z","body":"**[Test build #63255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63255/consoleFull)** for PR 13701 at commit [`3c7afaa`](https://github.com/apache/spark/commit/3c7afaa680aca0d508b6770d863bc164d2bb78bd)."}}
{"actor":{"display_login":"SparkQA"}}