{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11437","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11437/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11437/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11437/events","html_url":"https://github.com/apache/spark/pull/11437","id":137410070,"number":11437,"title":"[SPARK-13582] [SQL] defer dictionary decoding in parquet reader","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-02-29T23:17:50Z","updated_at":"2016-03-01T18:37:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11437","html_url":"https://github.com/apache/spark/pull/11437","diff_url":"https://github.com/apache/spark/pull/11437.diff","patch_url":"https://github.com/apache/spark/pull/11437.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR defer the resolution from a id of dictionary to value until the column is actually accessed (inside getInt/getLong), this is very useful for those columns and rows that are filtered out. It's also useful for binary type, we will not need to copy all the byte arrays.\r\n\r\nThis PR also change the underlying type for small decimal that could be fit within a Int, in order to use getInt() to lookup the value from IntDictionary.\r\n\r\n## How was this patch tested?\r\n\r\nManually test TPCDS Q7 with scale factor 10, saw about 30% improvements (after PR #11274). "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/190845510","html_url":"https://github.com/apache/spark/pull/11437#issuecomment-190845510","issue_url":"https://api.github.com/repos/apache/spark/issues/11437","id":190845510,"user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"created_at":"2016-03-01T18:37:09Z","updated_at":"2016-03-01T18:37:09Z","body":"@nongli There is no visible difference on all existing benchmarks (ColumnarBatch and ParquetRead), they don't use dictionary encoding.\r\n\r\nAfter changed the intStringScan to use dictionary encoding (small number unique values), here is the result:\r\n\r\nBefore this patch \r\n\r\n```\r\nIntel(R) Core(TM) i7-4558U CPU @ 2.80GHz\r\nInt and String Scan:                Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n-------------------------------------------------------------------------------------------\r\nSQL Parquet Reader                       1248 / 1281          8.4         119.0       1.0X\r\nSQL Parquet MR                           1962 / 2093          5.3         187.1       0.6X\r\nSQL Parquet Vectorized                    876 / 1018         12.0          83.5       1.4X\r\nParquetReader                             741 /  755         14.1          70.7       1.7X\r\n```\r\n\r\nAfter the patch \r\n```\r\nIntel(R) Core(TM) i7-4558U CPU @ 2.80GHz\r\nInt and String Scan:                Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n-------------------------------------------------------------------------------------------\r\nSQL Parquet Reader                       1247 / 1279          8.4         118.9       1.0X\r\nSQL Parquet MR                           1809 / 1851          5.8         172.5       0.7X\r\nSQL Parquet Vectorized                    805 /  909         13.0          76.8       1.5X\r\nParquetReader                             742 /  756         14.1          70.7       1.7X\r\n```\r\n\r\nWe can see 10% improvement on SQL Parquet Vectorized, but no difference on ParquetReader, I don't know why. (I didn't included #11274 )"}}